The rise and impact of adversarial machine learning
Getting into machine learning is all fine and well. And, yes, it’s something that has revolutionized businesses and started so many opportunities for better products and services. But, like with everything that brings good, there are threats to it. And in comes adversarial machine learning (AML), as a threat to machine learning and its outcomes. Since ML is a field that is constantly evolving, much is still unknown. And this presents a playground for possible future attacks on ML integrity and model accuracy. What actually is adversarial machine learning Adversarial machine learning (AML) is a technique used in machine learning mostly to manipulate ML models or fool and misguide them with malicious input. It’s the process of extracting information on ML systems to rig and modulate outcomes, which reduces the accuracy and influences the model’s performance. AML has been on the rise since ML started gaining momentum. Even now, we can’t talk about all the ways adversarial attacks can influence machine learning training and modeling since new ways of attacking are turning up. Machine learning and AI are not simple. Thus, each new form of attack on them will be harder and harder to identify and eliminate. When it gets to the attack In machine learning, there are two main types of attacks: black box attacks and white box attacks. Black box attack – This kind of attack happens when the attackers do not have all the information on the targeted model, its architecture, parameters, or inner workings. White box attack – This attack is the opposite of black box. Here the attackers have information about the model and architecture. It is obvious which ones are more easily performed and more dangerous than the others, but both present issues in machine learning progress. And companies that deal in machine learning are investing more and more into battling such threats. But, when we talk about attacks we can say that there are those that happen at the learning level and modeling level, meaning that one affects learning data (train time attacks) and the other after model deployment (inference attacks). Data poisoning or poisoning attacks These kinds of attacks influence the data during training time. Attackers infiltrate data by changing existing data or introducing incorrectly labeled data. The model will make incorrect predictions since it was fed wrong or corrupted data to train the model. The point of the attack is to reduce the accuracy of the model and its predictions. Evasion attacks These attacks directly impact data used in models. It refers to designing an input that seems normal to humans but is wrongly classified by machine learning models. Most often it happens with image processing where a layer of noise is added to the original image. So one image might seem fine to the human eye, but with the layer of noise, it’s actually different. Model stealing or extraction attacks This one as well focuses on the model after training. It’s oriented on either reconstructing the model, basically stealing the model structure, or extracting the data it was trained on. How to combat adversarial attacks? If one of the above scenarios worries you, as it should, there are certain methods to use in combating adversarial attacks. Let it be noticed, that with each day and each machine learning progress, new types of attacks emerge. So, it’s vital to stay on top of every point where your model could get affected by malicious intents. Adversarial training One could train the model to recognize adversarial attacks. It’s a supervised method that feeds adversarial examples into the model so it recognizes them as threats and prevents future attacks. Here, the model continuously learns about possible adversarial examples and tries to diminish their effects in the future. Switching models Here, you should use multiple models in your system, so it’s harder for the attackers to target one. Since the models are switching, the attacker won’t know which one is in use and where he should strike. The attacker could target all models, but it’s harder to poison all of them in comparison to poisoning one. Generalized models In this case, multiple models are used as well, but they are combined into one generalized one. Here, all models contribute to the final results. Attackers might hit one of the models, but it would be harder to get them all. Defensive distillation In distillation, you train one model to predict the outputs of another model that was trained on real data. Here we have a teacher model that’s trained on real data and a student model trained on teacher’s outputs. Often, student models are smaller and faster than teacher models. This approach to combating attacks needs less human intervention and is adaptable to unknown threats. How do adversarial attacks impact decision-making? If the model is attacked and final results or predictions are skewed, then it’s obvious that every subsequent decision will be wrongly based. Those decisions can range from having minor influences to catastrophic ones, depending on the model use case. If the decision is related to financial outcomes, the result will be a monetary loss. But it can also manifest in lost opportunities. You, as a decision-maker, expect certain outcomes from the machine learning model. When maliciously affected, those models don’t accurately present results. So, each insight is wrong and leads to making other related things wrong as well. Decision-makers need to work on accurate data and insights to create strategies based on true findings. If the model is oriented externally, to serve users, their experience will suffer and this will result in negative feedback or lost customers – in case the model’s outcomes have to serve those customers (recommendations, shopping experience, taxi services, delivery, etc.). Other issues are delays. If the model is affected it needs to be reworked. These kinds of setbacks delay decisions moving forwards or features being deployed. Whichever way you look at it, inferences derived from machine learning influence further actions. And their accuracy or relevancy defines the positivity
Feature store in machine learning – why you need it?
With the way machine learning unfolds, data used in testing and modeling must come at the best quality relevant to the learning process. From data cleaning to data labeling, the process can be tasking. It’s not only about creating a simple data pipeline, machine learning and data processing extend beyond that. Machine learning often isn’t a one-and-done deal. It’s used more and more for operational purposes and for creating data-driven applications. That’s why data teams turn towards creating or using a feature store. But first, what is a feature in machine learning? Basics of features in ML Data comes in raw formats. To use it in any data project, machine learning projects specifically, it needs to be transformed into features. Features are, simply put, inputs used in machine learning to perform predictions. Features in ML are independent variables or measurable pieces of information that classify the type of data used in machine learning. They are building blocks of making predictions, and for example, in some cases, they can be defined as age, gender, weight, and so on, depending on the data set and what it describes. Defining features is called feature engineering and it’s often a time-consuming process.There are two types of features: continuous and categorical. Continuous features These are numerical values, within a certain range, that represent detailed information on selected data. They are more precise, or accurate since they are defined by numbers. For example, weight, height, etc. Categorical features These are categorical values or information according to which data is divided. Basically, categorical data represents a specific category or class. For example, gender, other fixed data, etc. What about feature engineering? Feature engineering is the process of raw data extraction and transformation, and turning it into features used in ML. Simply, it’s the action of converting raw information into usable and useful features that describe those observations. It leverages data specifically for machine learning by creating variables that can provide more information and provide accuracy in ML training and models. The rise of feature store As mentioned above, feature engineering is time-consuming and not an easy task. The success and accuracy of machine learning predictions and insights depend on the quality of feature engineering. This is where feature stores come into play. A feature store is an ML-specific data system designed to store commonly used features. A feature store transforms data, stores and manages feature values, and retrieves data for training and inference. For machine learning predictions to work, models have to go through a training process based on historical data and prepared features. Once the model has been trained, new data inputs need to be, sort of, operationalized through pipelines that will transform new data in line with defined features from the training process. Image 1: Feature store system Components of a feature store Some standard components of a feature store are transformation, storage, serving, monitoring, and feature catalog. Transformation refers to data transforming from raw format into neat features. Feature stores need to perform transformations by defined definitions and rules. Storage is keeping features stored for future reuse. It’s a centralized depository for features that can be used across multiple different models. It consists of two databases, online and offline. Serving is oriented towards exactly that – serving features to models. Operational monitoring does what it says, it monitors operations inside a feature store. A feature catalog or feature registry is a repository of features based on definitions and metadata that anyone can easily find and use. Image 2: Feature store components How does it influence machine learning? Feature stores are used as centralized storage where commonly used features can be accessed and processed across multiple ML models, teams, and entities. But they are not only used as “storage”. They transform raw data from various sources into features used for training ML models. By minimizing the data engineering part used in data transformation, the feature store provides quicker training-to-model time and minimizes the effort levels in data preparation. For big companies that are dependent on machine learning to provide services or products, feature stores are a great way to handle large amounts of data processing for data training and modeling. Benefits of feature store Not always will you need a feature store. You will use it only if you have many models that share the same features so you don’t have to go through the feature engineering process multiple times. So, for such companies feature stores bring a lot of benefits. Feature reuse When features are developed, they can be stored for future use. This allows them to be shared among teams for any future project, which improves the speed of model delivery. Centralized data and features storage Feature store keeps features in one place which implies centralization. It’s a centralized platform for the development, storage, modification, and reuse of ML features. High model performance Feature stores have centralized feature pipelines with the same feature definition, so their implementation remains the same through training and inference to modeling. This speeds up the process from raw data processing to have data ready for machine learning since it abides by the same rules and definitions. Security and data governance Knowing which data was used in machine learning, benefits the data team when they have to go through iterations or solve issues. Feature store also solves the issues of sensitive data. If it’s not needed in the modeling, it can be cleaned and removed before it was put in the feature store. That way data scientists don’t have to worry about weeding out sensitive data from training. Enhanced collaboration and feature sharing Having a centralized feature store allows different teams to access data for their ML projects without interference. Various projects might require the usage of the same features, so such stores diminished the need for them to go through multiple levels of authorization to access data. Data teams can also share ideas, improve the speed of feature engineering, and solve possible issues. When do you need a
Synthetic data generation and why is it becoming popular
No one can stress enough how data is one of the most valuable resources these days, not only in business but in our daily lives as well. But like the real world, data also isn’t perfect. It’s hard and costly to collect data, and it comes with its own set of shortcomings, not to mention sensitivity and the complexity of its nature. For those reasons, synthetic data was introduced as an alternative to be used in training machine learning models. What is synthetic data? Synthetic data is information that is artificially generated. It’s not taken from real-world happenings, but it’s created with simulations and algorithms. And it’s used to test mathematical models and train data for machine learning methods. Its purpose is to mimic real-world observations and events. Synthetic data was introduced for multiple reasons, primarily to provide a reliable source of information for data modeling and minimize the costs and uncertainty that come with real data. It is used for testing applications, protecting sensitive data, training machine learning models, and validating systems at scale. Also, real data needs labeling which takes time, a lot of effort, and is costly. Synthetic data comes already labeled, and it’s labeled correctly. It was introduced also to fight data scarcity and unavailability. Most importantly, it contains no sensitive, private, or personal data points or values. Types of synthetic data Partial synthetic data This is a data set that includes synthetic data and real data from existing occurrences and data sets, but it omits sensitive information. Full synthetic data This type of synthetic data has no connections to real data and it’s fully synthetic. This means that all the required variables are available, yet the data is not identifiable. Hybrid synthetic data This data is partially synthetic and partially real. It contains both, sensitive data and synthetic values, but still provides security in handling data since values can’t be tracked back to the original source. But, if we look at data in detail, three more types of synthetic data can be identified based on the content and form: Synthetic text – Used mostly in natural language processing, synthetic text is great since it hides sensitive information.Synthetic media – This refers to media like videos or images, used mostly in object detection and real-world applications and recognition tasks. Synthetic tabular data – This type of synthetic data imitates real data for data science projects by providing data structured in rows and columns to fill in the missing values in real data. Benefits of synthetic data Synthetic data brings loads of benefits over real-world data, and some we have mentioned thought the post so far. But security, privacy, and reliability stand out the most. Higher quality data and format of the dataset – As we said before, synthetic data comes at a higher quality than real-world data because it diminishes the possibility of having bad data, missing values, or irregularities. It is created for the purpose to mimic real data but without the fuss of cleaning or labeling data. Faster project development – Collecting real data is time-consuming. Synthetic data speeds up the process and availability time of quality data, which in turn helps for faster development and a shorter time-to-market. Increased data privacy and security – Synthetic data means that private information isn’t connected to the real data source and can’t be re-engineered back to an individual. Lower costs – Creating synthetic data is far cheaper than collecting, cleaning, and transforming real data to fit certain criteria. Even then, the imperfections of such data are what can affect ML models. Let’s talk about GANs and VAEs Who doesn’t love a good abbreviation? Especially when it sounds important.Well, if you are diving into synthetic data, you would want to know how, images for example, are generated. Besides, autoregressive models which are dedicated to synthetic time series data, we have Generative adversarial networks (GANs) and Variational Autoencoders (VAEs). Generative adversarial networks (GANs) consist of two sub-models: generator and discriminator. The generator creates fake data, and the discriminator defines whether the data is fake or real. The discriminator is trained on real data to identify if new data inputs are real or fake. The generator then identifies more realistic data points that the discriminator won’t be able to identify as fake. It’s a sort of circular process where each model works against the other, one creates data until the other can’t distingue real from synthetic. Variational Autoencoders (VAEs) use GANs but with an additional encoder. They learn from data and use real data points to reconstruct them or to create additional variations. It is similar to real data in structure and characteristics, and it’s highly realistic. Parts of this model are the encoder and decoder. Why is synthetic data changing the ML and AI game? If you look at some of the biggest players in the data market, like Google, Meta, and other sites that collect huge amounts of real user data, they have ruled the business so far. This has discouraged small companies from getting into the race. But it all changed when synthetic data entered the game. It leveled up the playing field since no one will be the sole owner of such data. This started the process of democratizing access to data. It also stimulated innovations and AI development as we know it today. Besides the obvious benefits synthetic data brings, it offers even more possibilities for using it to progress ML and AI models. Certain data limitations that were here before, are getting slowly erased and new capabilities unravel. Imagine a scenario where you are not limited by data, issues, missing values, or bad data! Imagine what kind of ML and AI projects you could do with sufficient reliable data sets! Options are unlimited, and results are more accurate. Of course, you would want to access ML and AI based on real-world data that represents actual scenarios. That’s true. And you will. But for the purpose of training those models, you will need synthetic
How to successfully build data and AI teams
A lot of companies are becoming data-driven. Which is a fabulous thing since the world we live in is primarily revolving around data. Markets, businesses, and consumers, all drive decisions and actions based on information. But, purely recognizing the need for data is not enough. Becoming data-driven means either building your own data and AI teams or employing others. Either way, you are up for some challenges in building those teams. Hiring talent has become a bit difficult in terms of finding and attracting the right people with the right skills. Companies compete with each other for employees, so they push benefits and all the bright things to garner attention. That means that standing out among many others and becoming interesting to data scientists and data engineers, especially seniors, is a daunting task sometimes. How to approach recruiting in a new market dynamic? Well, this might be one of the hardest questions to answer. What is the most optimal way to attract talent and get them to work in your organization and data team? There is no perfect formula for this, but the best first step is to be authentic. Show that your company is somehow unique and share your expertise. You’ll have to distinguish yourself by becoming an expert in a field where your knowledge becomes attractive to others. Here is where employer branding comes into play as well. Your culture, mission, and vision show the purpose of work to those that find this important in their careers. This is the approach toward seniors. They are at the point in their life where they search for a higher purpose and work-life balance to fit into their everyday life. For mid positions, they search for career development and higher levels of responsibility. They are on the track to becoming seniors and career advancements are vital to them. For juniors, mentorships come as an integral part of work. Guidance and help are what they need in their pursuit of knowledge. Go beyond technology Data teams should focus on skills and experience rather than technology. With the way technology rapidly changes, new things have to be learned. And what you don’t know, you will work on mastering. Technical skills can be gained. People can shift focus and technologies so that other skills can come first. Preferring technology-agnostic skills is how you can build employees up, you have foundational knowledge on top of which you can build further skills. Problem-solving, leadership, communication skills, and willingness to take responsibility, are things that predetermine how much will you have to invest in an employee and how much are they willing to work on themselves. First thing first, you are hiring people, not robots. Each individual is more than just their technical skills. What you want to search for in great employees are soft skills, empathy, and a collaborative spirit. As we said, technologies can be learned, soft skills take time to acquire and depend on personal aspirations. So, if you can nail down such employees that have soft skills and want to learn, those are the kind you want to chase. But let’s keep it real. You can’t search for something you, yourself, can’t offer. Positive culture and atmosphere are what most in IT search for. Enabling an empowering and stimulating environment is a must. Don’t sell lies You can promise the moon and the stars, but if it’s not true, don’t do it. Being transparent is important. Set the expectations right and always be clear about projects and the work you expect from future employees. Even if you paint the prettiest picture ever, if it doesn’t hold up, you’ll end up with negative results. Manage expectations from both sides. It is better for the long run if both sides are completely honest since they all know exactly how things will flow. Trust us, even if that way you dismiss, or you are dismissed by some candidates, you’ll end up with a perfect match for your team. One great thing to do is introduce candidates to projects and technologies used in the company. Describe everything in detail so they’ll know exactly what will they work on. How technology dictates who you need in your team? Some companies specialize in certain technologies. And often projects dictate which technologies that will be used. So it’s no wonder that during the recruitment process, you will try to find people that fit that criteria and have knowledge of these technologies. But, when we are talking about projects that aren’t due soon, you can and should seek people who, even though have no extreme expertise in a specific technology, are willing to learn and deepen the skills they have. Let’s say someone has some basic knowledge and understanding of a particular field or technology. You can invest in such people and make them become masters in it. In this case, it’s up to you as an employer to provide the means and support for that person to become an expert. As we said, this relationship is two-sided. Also, think about how long will it take for an individual to learn and adapt to new technologies. What also influences your choice of employees are trends and predictions on how certain tech stacks will be attractive or necessary in the future. Is something going to be more desirable than the other? Will employees with knowledge of older technologies be harder to find if a project requires one? You have to think long-term and seek out what is sustainable. Can we talk about soft skills? Anything can be taught, right? Well, yes, but some skills have to be ingrained in you. Soft skills are the way how you communicate with others. People aren’t tasks you can mark off when done. Compassion, integrity, and understanding are things you have to have when working with other individuals. You need to know how to listen to them, take their advice, and collaborate. Managing a team or even working in groups is not easy. You have
When is data fabric great for your business?
In a previous post, we had an introduction to the data fabric concept and architecture. Renato covered the topic perfectly and outlined the benefits and why should you embrace implementing it into your business. It might seem like a new buzzword, but in earnest, it shifts the trends in data analytics and data management. But, let’s recap what data fabric is. Gartner defines data fabric as an emerging data management design that enables augmented data integration and sharing across heterogeneous data sources and emerging data management design for attaining flexible, reusable, and augmented data integration pipelines, services, and semantics. A data fabric supports both operational and analytics use cases delivered across multiple deployment and orchestration platforms and processes. Or as Renato mentioned, it’s a data architecture that integrates a set of technologies and services designed to achieve the ultimate objective of data democratization and self-service across the enterprise. Simply said, data fabrics are systems that allow for data to move seamlessly around different parts or applications. If we want to get general and even simpler, data fabric is just that – a fabric that is woven and structured from threads connected into one cohesive entity. Just as threads connect, the same way data systems, applications, and architecture, are all interlaced together to ensure smoother and unobstructed data flow, without the need to have one centralized data repository. Let’s track what your business gains Where does the value kick in? Well, anything that ensures quicker operations and data flow is bound to bring some major benefits. If everything moves faster it means it’s more cost-effective. But it also means that people inside the company can do their work more efficiently and in less time. If that in itself isn’t attractive, data fabric brings loads of other benefits. There is a reason why, recently, data fabrics gained even more popularity. In the previous post, we concluded that data fabric brings these benefits: But, we’re going to expand a bit and turn our focus on perhaps more of a business end of a specter. Improved data access and integration If you get the gist of data fabric, you know that through data virtualization and data catalogs, it provides an overview of the whole data environment. It acts as an augmented layer over each data source and storage and gives access to data without the need to move data from one point to the other. By giving all-around access to information, we can say that data fabrics improve and enhance data democratization. Non-technical personnel can now work with data in real time and avoid waiting for data teams to provide data back to them. This way companies can shorten the time to get the right data to the right people. Data catalogs help in giving the information where they can find certain data, so they can far more easily navigate such environments. One great thing about data fabrics is that any change in data is automatically registered and handled. Data fabrics kind of learn and adapt to changes, and, let’s say, sort new information where it belongs. With such a high level of accessibility and integration across the whole company, there is increased knowledge among employees. Their expertise and skills rise, their confidence increases, and they can feel more comfortable making data-based decisions. If you can access data at any point, you can react more quickly to changes and minimize or even negate negative effects. Increased agility As we said before, data fabrics are agile and non-rigid. As data environments evolve, data fabrics follow suit and learn new changes to which they adapt. If you are familiar with the term data agility, then you surely know what that entails. Fast and efficient data movement removes the obstacles of rigid or siloed data, while simultaneously providing better control. As established, data fabrics provide an additional layer of flexibility where each data point is connected and can travel across applications or components without obstacles. But, in terms of business value, agility is not only a description of data but the agility of employees and business processes or operations. People using and working with data become more agile to its changes and meaning. With data democratization in tow, data becomes a great asset utilized by everyone. This is where you don’t have to wait a long time to get to information, but rather you can access it instantly. Your workflow instantly changes with this. Your tasks get more easily achievable and accurate. Faster decision-making It’s no surprise that with increased agility comes better and faster decision-making. They go hand in hand. Data fabrics also make way for better business insights and a single source of truth. A 360-degree view of data results in faster value delivery, whether we talk about operational changes or even deploying new products or services to the market. With an unobstructed data flow comes a flexible approach to making decisions. You can imagine data fabric as interlaced highways and speed lanes where you can go faster and more smoothly than on regular roads. There are no complex intersections with stoplights that halt your progress. In work, you get instant input, analyze it, and drive decisions. Increased efficiency and innovation momentum Okay, now that we have established that data fabrics bring complete overview and access to data, we can safely say that they can increase efficiency and stimulate innovations. Simple as that. Before, employees or stakeholders were at a standstill because of the lack of available information. Even if you wanted to analyze or access certain data sets that could give you insights into products or customers, there was sort of a bureaucratic stasis. An augmented layer for data removes that. Data fabrics are also designed for and can act as stimulants for AI and machine learning. These innovations are driving the market right now, and using them internally can only maximize your company, product, or service value. Speeding up the innovation momentum gives leverage and competitive advantage on the market, and even marks
How to tackle data security and concerns
In these days of massive data amounts, AI, the development of online media, and such, data privacy and personal data have become an all-time concern. Alongside people that try to test the boundaries of data security and put such information at harm, even data systems and infrastructure can show weaknesses in their defenses. But what is data security? Data security is the process of protecting data from unauthorized and harmful access, corruption, or theft throughout its whole lifecycle. It means protecting data from any infractions and perilous actions, cyber-attacks, or data breaches. Data security can be seen through data management processes, software and data infrastructure, storage, and data utilization, but also through data policies and procedures. Digital transformation has changed industries and will continue to do so. But with each passing step, security and data concerns have become strenuously important. The volume of data is growing exponentially, and it can prove to be challenging to handle and protect. But, not only companies are aware of data endangerment. Consumers and other users show arise in awareness about data privacy and security issues. Even with laws in place, they are still wary of handing out private data and giving others access to it, and rightfully so. Not everything is as easy Data security might seem like a pretty straightforward concept, but it faces many challenges along the way. Each day brings new threats and areas where weaknesses can be exploited. It’s not only external dangers we have to think about, but internal ones as well. Those might be even more complex than we think. Data storage A lot of companies save their data in cloud storage because of the sheer volume of data. This means that the control of access can’t fall in the wrong hands. Smaller discrepancies can cause someone not to get access, or give it to someone who’s not supposed to have it. How companies store data is important because of the structure, access, and security access validation processes. Fake data When companies use external data sources, it’s integral to validate those data sources and discard fake data created to mislead or misinterpret results. Fake data can skew results and take precious time and resources to get discovered. Continuous data validation should be implemented to prevent anomalies and false findings. Data privacy Data privacy is one of the biggest challenges in today’s world. Personal data has become somewhat of a hot topic. Preventing security risks in data privacy is not only regulated by law but is of the highest importance. Personal data and such, are viable for cyber-attacks which can bring a lot of damage in terms of privacy breaches, loss of data, or criminal activities with said data. Data management How you manage data is also how you’re going to protect it. Managing user access is one part of it. Structural data management is how companies make sure the infrastructure is done correctly with minimized risk of data infractions. Implementing data observability can help prevent breakage or mistakes in those data management procedures. Data access control Controlling which users can see which data, helps preserve data integrity and privacy. It kind of goes against data democratization, but it provides a higher level of security so unauthorized people won’t access sensitive data. Data poisoning Data poisoning involves tampering with machine learning training data, by infiltrating maliciously crafted samples, to produce undesirable outcomes. Especially, today when we have so many AI and machine learning tools, it’s tricky to maintain security if only one breach can influence the outcome of the machine learning models. Employee theft Trust is a valuable thing, and having trustworthy employees is even more valuable. But, one can not be always sure that some might not misuse data to do harm. With data democratization on the rise, it’s becoming harder to control who has access to which data and how they use it. How to ensure data security? Data security is an interesting and really important topic. It’s not something it will ever be taken lightly. With so many risks and threats, companies invest a lot into data protection and making sure it stays safe in their hands. From various procedures to policies and regulations in place, we can sum up some of the data security controls into these segments: Transparency and compliance There are certain laws in place, such as GDPR, that serve as a way to protect data and privacy. It’s there to define how customer data can be handled. It also puts the power in the consumers’ hands who decide how they want to share their data. Implementing data transparency and making sure all employees on all levels are familiar with those rules and regulations is integral for stable data security processes. Access controls As mentioned before, regulating who has access to data is the first and foremost important step in securing data. As we said, data democratization is on the rise, but not each piece of information should be available across all levels and users. Someone from the lowest hierarchical level doesn’t necessarily need to know top-level information and maybe vice versa. The same goes for different departments. Yes, they might need to share some data that is connected to them both, but certain aspects can be kept to individual users. Also, controlled access to devices or server rooms is an important physical security control. Authentication Multi-factor authentication or verification is important to keep tight security in place. This defines how a system verifies user identities before granting access, from password protections to biometric authentication. Multiple user verification steps are a way of making it harder for other users to access data and content they shouldn’t see. Backups and recovery In the case of data corruption, loss, or system malfunction, having backups and data recovery processes in place is a way of making sure that data is not completely lost or damaged. Secure data backups help companies gain control of their data in case of ransomware or other security threats. Data erasure Sometimes
Get to know our data detectives – Wilim
Come and meet Wilim, our data detective and data engineer. With a name so unique, you can bet his data skills are also unique. Dive deep into what he loves about his works, what motivates him, and why he believes data engineering brings so much fun and creativity. Tell us who you are and what you do in Digital Poirots. Hi dear Reader, my name is Wilim, and I officially work here at Digital Poirots as a Data Engineer, but, if you ask me what I unofficially do here at Digital Poirots, my answer would be that I am playing with magic over the data like if I was David Copperfield. What does your typical day look like? I always start my day with a cup of coffee while reading e-mails or scrolling through unread messages on my Slack if there are any. After my morning routine, I spend the rest of my day working on tasks assigned to me, in collaboration with my teammates, and with one more cup of coffee. What made you decide to develop your career in this field? I think the main reason was the way of thinking during the programming – you often have to be imaginative, creative, and predictable in doing that. Additionally, every day brings new challenges that push you to learn something new. What drives you in your work? Music and coffee! What’s the best part about your position? The technologies I use at any given time depend on the task or project I am working on – a new task or project may imply new technologies, but the concepts will still be the same. So that’s the best part of my job because I don’t have to worry that one day I’ll get bored with my job because of working with the same technology all the time. What don’t you like about your job/work? Writing the documentation. What’s the biggest mistake you’ve made in your career or what ups moment you had? Right now, I can’t remember any big mistakes I’ve made, but of course, there surely were many smaller mistakes I’ve made along the way. What drives you crazy about your job or your daily activities? Multitasking drives me crazy. It can add extra time to the task I’m working on, and if it happens often it makes me tired. Also, some unpredicted circumstances related to the task may have the same effect. Which technology/tech stack do you like the most? The tech stack I like the most is the data engineering-related stack, which contains technologies such as Python, SQL, ETL tools (e.g. SSIS), Apache Kafka, and CI/CD tools (e.g. Jenkins). What advice would you give to someone entering this field? I would tell them to enjoy the wonderful data universe where every data problem is unique and has its own approach to solving it. That pushes you to come up with creative solutions and makes your job more exciting. Do you have any funny or interesting stories that happened here in the company? My employers threw a surprise party for my colleague Ivo and me after graduation. The surprise included a cake, which was even more significant for me because it was the Ledeni vjetar, my favorite cake. Your favorite person to work with? I like to work with all members of Digital Poirot’s team (data team), especially our common discussions on a glass wall in the office (that we use as a writing board) related to data, of course. If you can compare your job/work to one movie or show, what would it be? Pat & Mat, I would say that sometimes data engineering, especially a part of data engineering that is close to the DevOps domain, is not always rainbows and butterflies. We sometimes face challenging infrastructural problems where searching for a solution can seem like we use a trial and error method as if we were in Pat & Mat. If you can choose one song to go along with your job or which would make you be really in the zone while working, what would it be? Something from Gigi D’Agostino always keeps me in the zone while I am working, especially the songs La Passion and L’Amour Toujours.
Data ethics and why it’s important for brand trustworthiness
There isn’t a blog post nowadays without someone saying this: “All companies are data companies.” And it’s absolutely true. Data has become the pillar of business society and something people trade with. Information in the digital age has become somewhat of a gold mine. But, with each piece of information comes the hot topic of data ethics. Data provides a lot. All the opportunities are almost unfathomable. But, each process of this data utilization comes at a price, a price for companies, and a price for consumers. Especially in today’s world, we provide so much information about ourselves and often don’t think about what happens with that data. We buy services and products where we are obliged to provide certain information and we as consumers take that as a necessary step to acquire those services or products. What do data ethics entail Data ethics focuses on the ethical exploration of data in terms of collecting, generating, analyzing, and using data, whether personal or business. It proposes a set of rules and guidances to manage data in a manner that lies on ethical foundations. Gartner defines “data ethics” as a system of values and moral principles related to the responsible collection, use, and sharing of data. With data collection and personal information, it is integral to comply with moral and ethical rules so the customer who shared said data feels comfortable with his data usage. Transparency is key, and each person should know what his data is being utilized for and how. It might seem that the market and all the data are free-for-all kinds of environments, but we couldn’t be more wrong. Besides governmental laws and regulations which protect data, consumers are more and more educated on how their personal information is being used. Consumers ask more questions and sometimes they can be wary of which information are they willing to share. Data analysts, scientists, and engineers hold data ethics in the highest regard and they have to be careful how they handle data. This has shaped how companies behave as well. With data democratization being a frontrunner in data trends, they need to carefully compose rules and foster a culture that is oriented towards data ethics. Companies with a well-defined ethics culture can react easier to data mishaps or misuse. For example, in case of data leakage, they can respond much faster and communicate with consumers about what happened and how they are going to fix it. This, in turn, creates a bigger level of trust in the company. Data ethics culture can also be a great technique for creating strong brand awareness and trustworthiness. Core data ethics principles Individual control of data Each person is the owner of their own data and this must be established early on. Private data belongs to the person it originated from. They choose when and how they want to distribute and share that information. At each point, consumers or users have to allow others to use their data, by consenting to policies, cookies, terms and conditions, and so on. It is unlawful and unethical to use someone’s data without their consent and without them being informed of it. Transparency Each person has the right to know how their data is being used. Companies need to be transparent with their methods and intentions. People, whose data is the subject, need to have the right to decline to share their data if they think it will be misused. Privacy Data is private and people who share that information must be ensured of their privacy. Countries have laws that protect that privacy. Each person must be informed if their data is being sold to third parties. In most cases, data must be anonymized and not connected to an individual person. Data can be used to describe patterns and trends, but it shouldn’t be linked to one specific person. For companies, it is integral that data is stored in secured databases and that no one, without authorization, can access it. Accountability Accountability refers to the company’s sustainable and ethical use of data. Those are the foundations on which data utilization is built. It’s how companies respond to data protection and which tools, rules, and guidelines they have implemented in order to achieve that data privacy. Equality Often, in data analysis and methods and algorithms implementation, we can encounter unintentional bias. Sometimes, personal data is used based on predefined notions and stereotypes. This is why there must not be any ethnical, racial, or gender biases. If data is used to personally target consumers, it must not be based on the above-mentioned characteristics. This is why the data ethics cultural board needs to be diverse and inclusive. How do data ethics build brand value and trustworthiness? If a company has built a great data ethics culture, it can be transferred to the brand value. Building strategies around the creation of a safe data environment and ensuring that customers are aware of that can increase the trust in that brand. Data privacy policies and ethical guidelines can promote the company as highly secure and respectful to consumers and their personal data. Even employing data scientists and data engineers who work diligently to ensure data protection, is a step in the right direction. Such services go beyond everything to ensure that data is utilized properly and in line with the company rules and regulations. But what can we outline as the biggest benefits of establishing data ethics and how a company handles data? A more positive brand perception Companies or brands that can brag about their extensive and stable data ethics culture, can create a more positive perception in the eyes of a broader public. If they are known as a secure company that doesn’t misuse or endanger private information, customers would be more inclined to share that positive perception with others. Bigger conversion rate and new customer acquisition A trustworthy brand can more easily attract or convert new customers. It can also sell additional products or services to
Conversational analytics and everything you need to know about it
With how the trends have been behaving lately, a lot of the spotlight was directed toward artificial intelligence and its capability to converse with humans. From AI-based chatbots and AI content generators to others, we have seen technology use available data to interact with others. But, it’s one thing to use data available through databases or some devices. What happens when you need actual human data derived from their interaction with other people? This is where conversational analytics comes into play. It uses technology to analyze speech and conversations among people or agents. Conversational analytics is the process of using technology, artificial intelligence (AI), Natural Language Processing (NLP), and machine learning techniques to convert natural language conversations into a machine-readable format. But, let’s not limit it only to conversations. It also gathers data from posts, comments, and similar. It draws insights from any interaction people make, in real life or online. How does it work? Conversational analytics is used to draw insights about customers, their behavior, and how they interact with a certain product or service. Its main purpose is to fully understand customer motivations and wants. Most businesses use it to get vital information based on which they can take instant actions in reaction to issues or previously unidentified customer needs. It depends on the usage of AI to convert natural language conversations into formats applicable for further analysis and machine learning methods. The analysis finds patterns, trends, or anomalies among consumers and even identifies root causes for dissatisfaction or customer churn. Some components of conversational analytics include: Text analytics Through natural language processing (NLP), conversational analytics can analyze text from chat, email, or social media. It analyzes common denominators or certain patterns to collect insights and present them through visualizations. It allows businesses to delve deeper into customer opinions and views on products or services. Speech analytics This analytics recognizes and turns speech into text format, to be analyzed through the same methods as text analytics. Why is this important? Well, if you have call centers or you want to collect data from videos or recordings of sorts, you need methods to analyze that data without having to manually transcribe it. Voice analytics This type of analysis explores how was something said and not what was said. It discovers the changes in someone’s voice, for example, if the speed of talking changes and if the voice gets louder or not. It serves as a great way to determine if perhaps customers get agitated or frustrated. The methods used here are different from speech analysis, but it still uses AI to detect those voice changes. Sentiment analytics One can recognize specific keywords or patterns in conversations through sentiment analytics. Each word used can mean a different emotion or sentiment towards a product or service. It uses similar methods as in text and speech analytics to discover certain words and phrases which describe an emotion or opinions. Benefits and possibilities of conversational analytics With the ability to convert conversations into actionable insights, we can clearly see the multiple benefits conversational analytics can bring to the game. Imagine all the possibilities for marketing and sales if they can act on that information instantaneously. Previously, the only available data was from static text and databases, but now with speech recognition, those insights convey a broader meaning and value. Data can be collected from chats, social media comments, phone calls, emails, tweets, shopping and business reviews, and many others. This generates so many points of data entry, that it can provide more detailed information about customer journeys and their interactions with the company or brand. Getting the whole story Some customer insights are limited and do not convey the whole message. Some people are freer to articulate their wants, needs, or problem by talking, rather than writing them down. By including speech, text, voice, and sentiment analysis all in one, it provides a much broader picture of customers and their behavior. Time-saving If there is a tool for conversational analytics, there won’t be a need for manual intervention and manual transcribing of calls or other voice records. This saves time, not only in data preparation but also in terms of creating valuable insight or metrics much-needed in real-time. This can make a difference in reacting timely to customer pain points or issues. Upselling and creating great leads and conversions If we can monitor what customers say or want at the exact moment, this can create an opportunity for upselling or a bigger conversion rate. For example, if a person contacts customer service asking questions about a product, they can target that person with an email about that same product and a call-to-action that leads to a direct purchase. This way, that caller can go from the first point of contact or inquiry to purchase in just one click. Increasing customer satisfaction and creating a better customer experience If companies can identify customer needs and wants accurately, the products and services they provided can be optimized and adapted to meet those exact needs. Even, being more efficient in answering customers’ questions leads to bigger satisfaction and a much better customer experience. Reducing operational costs By eliminating the need for manual conversation or speech transcribing, companies can reduce the cost of labor and time spent on those tasks. But also, by reacting on time to customer demands or dissatisfaction, one can reduce costs and possible losses. Quicker and more effective damage control If a customer expresses their issue or pain point, through customer service or even through social media or business reviews, conversational analytics allows companies to instantly react to those issues and minimize the negative influence. This way they identify the pain point and can solve it in no time, which leads to lower costs in the long run. Imagine if a customer complains about a certain product attribute or difficulty on social media. Companies can react to that, take notice, and try to resolve it, so they won’t have any future complaints
Data observability in a nutshell
There is a lot of talk about broad aspects of data science and data engineering, but few mention the importance of quality data and the processes behind it. Data observability is a term that handles data behind the scenes. From infrastructure to data movement, it’s integral to provide an unobstructed flow of information. Data observability helps understand and manage data health, movement, and quality. It uses a wide spread of technologies and activities that allow you to identify data issues and bottlenecks in real time and prevent data downtime. When we look at complex infrastructures and data architecture, we can easily see why there needs to be a discipline in managing data across all data management tools, technologies, and organizations. Why data observability? You might think that if you have quality infrastructure and data collection tools in place, you won’t need to monitor data as often. That might be true in some cases, but with the speed and volume of data these days, issues might arise sooner than you think. Data observability allows teams to continuously monitor their systems and analyze how data interacts with all aspects of the IT infrastructure. This helps them to identify errors and issues they weren’t aware of and apply improvements to stimulate more effective data traffic. By monitoring data, teams across the whole organization can optimize processes and have set in procedures that simplify future data science and data engineering projects. Data observability leads to faster mean time to detection (MTTD) and mean time to resolution (MTTR) when issues occur. But, let’s face it, most organizations fail to turn their focus to data and how it moves. Going with predefined notions and thinking that data needs only monitoring is a recipe for failure in the long run. If it works now just means that it might not work in the future. With the way technology changes and how IT infrastructures change shape, most forget to, let’s say, update their data structure and pipelines. The pillars and basics of data observability For data observability to work, there are 5 pillars that contribute to its process, as defined by Moses: Freshness This applies to the freshness of data and how up-to-date it is. It prevents data from becoming stale, which could lead to wrong metrics, insights, and conclusions. Basing your decisions on old data leads to mistakes and loss of time and money. Quality It’s no secret that what you do with data and what you get from it, depends on the quality. All the infrastructure might be in working order, but that doesn’t guarantee that the data itself is good. The quality pillar makes sure that your data can be trusted and therefore utilized properly. Volume Data tends to go from 0 to 100 quickly, meaning that the volume and speed of data can change instantaneously. Data observability gives you those indicators that the volume has changed and that once low data quantity has become bigger. Schema Changes in data schema and how it is organized can often result in broken data and cause data downtime. It’s important to be aware of these changes in time, so companies can prevent data obstructions and issues. Lineage In case of data breakage, you want to know where it happens. Data lineage answers that and tells you which upstream sources or downstream ingestors were impacted and where the data came from, i.e who controls it. The data lineage process pinpoints exactly where the problem is when data breaks. It also collects information about said data or metadata which helps with governance. Benefits of data observability When we talk about data observability, some benefits can be summed up in these: Discovery of data errors and issues before they occur Data observability is here to ensure that errors and issues in data and its infrastructure can be detected before time. Being one step ahead allows for data to move more swiftly and smoothly. It also makes sure that bad data is detected on time before it even has the chance to end up in your data warehouse, for example. Minimized data downtime and a more stable data environment With data observability, one can minimize the time spent detecting and fixing errors. With the usage of machine learning models, environments can be learned and made more stable. This means that, in the future, responses to changes can be quicker and more effective. Timely delivery of high-quality data to all users across different organizational levels This means that data is continuously observed so the quality of it is much higher and more easily distributed across different organizational levels. With a more stable data infrastructure, it’s far easier to give access to data to more users. This is where data democratization comes into play. With a higher data quality, insights and decisions made are therefore more quality and accurate. Troubleshooting and resolving issues faster and on time Observability allows for quicker eros and issues detection. If there is a breakage in the data flow, it’s will be faster to identify where it happened and how so recuperating processes can start on time. Having a complete overview of data flow and architecture means a much better response time in any troubleshooting efforts. Greater operational efficiency and more efficient monitoring and alerting Data observability provides a way for data monitoring and alerting when changes occur. If there are infrastructures and monitoring tools set in place, operations can consequently be smoother and more efficient. If everything works and issues can be detected on time, all other operations will have less chance to encounter errors. Increased collaboration between different departments and roles We already established that data democratization and data observability go hand in hand. If data, and the right one, has an interrupted flow among departments, teams and people can collaborate more efficiently and exchange findings that matter. They can trust data that serves as a base for their daily operations. Data transparency is important when different teams have to make joint decisions. Enhanced trust