Visualization & streaming: How to use it to your advantage
Importance of data visualization Many people would say that knowledge sharing is one of the noblest things any human being can do. Aside from helping other people grow and be better, by sharing their knowledge people become happier, develop new professional or private connections and bring more purpose to their life. This is why the author of this blog post has always admired teachers and professors who are the most obvious examples of knowledge sharing. In a way, being a data scientist is all about knowledge sharing as well. Put in simple terms, data scientists extract the knowledge from raw data and share it with the world. Knowledge sharing is certainly a complex concept and it can be described as anything but a universal and straightforward process. There are many different theories as to how children and people, in general, learn the easiest and the quickest. While the process undoubtedly varies from individual to individual, there is an idea that the use of visualizations in the learning process helps students to learn faster and better. Check out the following articles for a word or two about this: The Use of Visualization in Teaching and Learning Process for Developing Critical Thinking of Students and Enhancing Learning with Visualization Techniques. Speaking from our own experience, visualizations definitely bring more fun and diversity to the learning process. We would also say that they leave stronger traces in our brains when compared to the usual learning methods, which is why the resulting conclusions are often easier to access afterward. Humans are highly visual creatures, sight is the sense they rely on the most and it brings them the largest share of information about the world around them. Because of this, it makes sense that interesting and concise visualizations could speed up the learning process. Also, it’s very important to keep the visualizations as simple as possible, the idea is to let them tell the story on their own, not to confuse the observers with unnecessary and redundant information. This is very well summed up in a quote by Ben Shneiderman: “The purpose of visualization is insight, not pictures”. Now that we’ve stated how important visualization is, it’s time to introduce some context to this blog post. This is the newest post in the series about our Retail Business Intelligence Platform project. After finishing up the streaming part of this project, about which you can read in our posts about Kafka and Spark, the next step was to generate appealing visuals in order to present our findings. We decided to go with two widely used visualization tools: Tableau and PowerBI. But before we show you some visualizations, a few words need to be said about data visualization in general. About data visualization Data visualization can be more or less effective, depending on the results and types of visuals used. To get the most out of it, one should know which visual should be used in which situation. Also, it is important to note that one of the main perks of visualization is the opportunity to easily send a desired message to someone without the use of words. As we all know, words don’t always come easy and can be tiresome compared to nice colorful visuals. However, sending a message with the help of visualization tools is anything but easy. In order to do it almost flawlessly, one would need to perfectly understand the data, the results, the human mind, and the laws of perception. Many factors should be taken into consideration when constructing visuals and, naturally, time and thought that has to be invested in the process grows with the complexity of visuals and the intricacy of the message one’s trying to send. If you’re one for more looking at visuals than coming up with them, you need to be aware of how easily you can be manipulated by someone showing you only visuals that support his or her agenda. This shouldn’t surprise anyone who knows that statistics and data science are closely related, considering that statistics itself is very prone to manipulation. Famous American novelist Mark Twain has popularized a quote about the manipulative nature of statistics and attributed it to the Victorian era British prime minister Benjamin Disraeli. Benjamin has allegedly once said there are three kinds of lies: “lies, damned lies, and statistics.” This is a peculiarly sensitive subject as manipulation and lying don’t always have to be deliberate. Statisticians and data scientists can be subconsciously biased and can affect the final results and visuals unintentionally. On the other hand, for those who create visuals one of the key pieces of advice is to keep it simple. People can get lost in trying to say too much with a simple visual, which can result in the viewer understanding nothing or getting a message that wasn’t intended. Of course, it’s not easy to make things simple and it takes a lot of experience and understanding. But it’s definitely worth all the effort because obeying simplicity keeps you from going astray by overcomplicating. Interactivity is very welcome as well. It changes the role of the viewer from a passive observer to an active participant. Essentially, it turns the process into an experience for viewers which makes it much more memorable and easy to understand. Tableau examples Just a quick reminder, our project included the unit sales data of more than three thousand products in ten different Walmart stores throughout three different states. We’ve tried many different visuals and charts and here we offer you a glimpse of the few most interesting ones. Figure 1 shows the 5 items with the highest average price for each department. Figure 1: Top Five Most Expensive Items for Each Department Figure 2 gives you two pieces of information at once. It shows the number of items sold and the total revenue for a certain item. Here we’ve picked item HOBBIES_1_345. Figure 2: Total Items Sold and Total Revenue for item HOBBIES_1_345 The next visual is Figure 3
Apache Spark – How to create a powerful streaming application
After setting up Apache Kafka, the next step in our Retail Business Intelligence Platform project was to set up Spark, another widely used software solution from the Apache workshop. Apache Spark, the real spark necessary to ignite our project and turn it into a true stream processing arrangement, is probably the most famous streaming engine available in the modern IT world. Spark was initially developed at the University of California, Berkeley in 2009 as a research project and, as years went by, it has grown into an open-source project that is developed and maintained by a large and collaborative community. Infrastructure Just like all other components of this project, Spark has also been configured to run inside Amazon Elastic Compute Cloud. We have created two EC2 instances, both of m5a.2xlarge type: one that serves as a master node and one that serves as a worker node. It’s important to mention that the default amount of storage on the worker node wasn’t enough, we had to increase it to 40 GiB. Of course, the optimal amount of storage is different for each individual project and it’s often very hard to estimate it beforehand. We recommend starting with the default amount of storage and, if necessary, upgrading it later. Setup Before configuring Spark to run on EC2 instances, we had configured it for testing purposes by installing it locally, which was pretty simple and straightforward. It is enough to download and extract Spark to a folder, install Java and, given that we are writing our code in Python, install PySpark via pip package installer. After that, all that is left to do is to run the job using the following command: spark-submit –packages <NECESSARY_PACKAGE_NAMES_SEPARATED_BY_COLON> <PATH_TO_THE_PROGRAM> Back to the real stuff, let’s talk a little more about our in-cloud setup. First of all, when working with EC2 instances, one should be aware of the differences and possibilities each of the different AMIs comes with. For example, we explicitly needed to work with Python 3.9, but Amazon Linux 2 AMI, the default AMI for EC2 instances, at the moment of writing this blog post comes with Python 3.7 preinstalled. One would think: ‘Piece of cake, I just need to upgrade the Python version from 3.7 to 3.9, that’s a few minutes of work.’ Well, unfortunately, because of a lot of factors that aren’t related to the main theme of this blog post which is why they won’t be discussed any further, it turns out it isn’t that simple if wanted to be done properly. Although it is possible, we strongly recommend you, if in need of Python 3.9, simply choose another AMI. For instance, Red Hat AMI doesn’t come with Python pre-installed at all, which is why it’s simple to manually install whatever version is necessary (sudo yum install python39 in this case). Just as we installed Spark, Java, and PySpark on our local machine, we had to do it all again for Spark EC2 instances. Furthermore, it was necessary to generate an SSH key pair in order to use launch scripts in Spark’s sbin directory. The public key had to be copied to each worker node in the cluster. Finally, in order to use Spark’s launch scripts, we needed to edit the conf/slaves file on master node and fill in the workers’ hostnames. After everything mentioned is taken care of, the cluster should be started by running start-all.sh script in the sbin directory. However, as PySpark does not offer cluster mode, but only client mode, the job has to be executed in the client mode from one of the machines inside the cluster. We confirmed this with a little experiment. What we did was execute the job from our local PC and, as expected, the driver couldn’t communicate with the cluster. The thing is that the driver process runs on the machine that executed the spark-submit command. Generally, after the driver is started, it tries to communicate with nodes inside the cluster. Firstly with the master node in order to get an allocation of workers, and secondly with those workers. It is crucial that both master and workers can reach the driver, and vice versa. In this case, they couldn’t communicate with the driver that was being run inside our local network. The IP address shown in Figure 1 confirms that the driver was indeed inside our local network. Figure 1: Screenshot of a Spark Application UI To conclude, the job had to be run from either master or worker nodes in the client mode in order for everything to work smoothly. We had chosen to run it from the master node and that is why we installed Git on our master node. This allows us to keep the code up to date efficiently and effortlessly. At last, we’ve arrived at the final piece of the puzzle, a command to get the job going: spark-submit –master spark://<HOST:PORT> –packages <NECESSARY_PACKAGE_NAMES_SEPARATED_BY_COLON> –py-files <PATH_TO_THE_PYTHON_LIBRARIES> <PATH_TO_THE_PROGRAM> Port for the cluster master is 7077 by default. Also, –py-files argument is here to allow us to distribute required Python libraries and dependencies.In order to prevent the unwanted termination of your job in case SSH connection suddenly closes, add & at the end of the command. Our use case If you don’t follow the thread that our project is, here’s a little recap of what had been done until Spark came into play. We handled preliminary data analysis, datasource preparation, and architecture planning. Also, we set up Kafka in order to store our messages into different topics and partitions. Now the time has come to take the raw data stored inside Kafka and turn it into some useful insights. The sales data we’re dealing with could be transformed into various key performance indicators, famously abbreviated as KPIs, that could help us gain a much better understanding of what is really going on in the world of retail. We are storing them in Kafka because that will allow us to easily implement real-time notifications later in the project. Apart from
How to setup a Kafka Streaming project
If you’ve read the previous blog post about this project, you should already know what was the main idea of the project. We’re trying to estimate the unit sales of more than 3000 distinct items in 10 different Walmart stores. The dataset provides us with more than 5 years of data about the dates on which the product was sold, the price of each product sold, and historical daily unit sales. Our idea is to simulate a real-life scenario by streaming the sales data. That’s where Kafka comes in. What is Apache Kafka? Apache Kafka is a very popular open-source platform used for event storing and streaming. It is basically a system with two main roles: producers who create and publish new messages consumers who subscribe to and read new messages Additionally, storing and processing data streams are tasks that can be and are handled by Kafka, as well. In Kafka, a server is called a broker. It receives and stores messages from producers to the disk and fetches and sends messages from the disk to consumers. Multiple brokers can constitute a group of brokers, also known as a cluster. Kafka messages are arranged into topics and each topic is furthermore broken down into partitions. To illustrate, a topic can be described as a bookcase, a partition is then represented by each individual shelf inside the bookcase, and books, of course, imitate messages. Infrastructure We are running all components of the project on virtual servers inside Amazon Elastic Compute Cloud, one of Amazon Web Services. These virtual servers are called EC2 instances. There is a large number of instance types one can choose while combining various CPU, memory, storage, and networking capacity options. We picked an m5a.2xlarge instance running on Amazon Linux 2 Kernel 5.10 AMI. It provides us with 32 GiB of Memory and 8 vCPUs. Docker is a very popular software platform designed to facilitate and accelerate building, testing, and deploying applications. It allows one to create an environment called container which makes it easier to run the code remotely or on multiple machines. All services we use for data streaming are packed in their own Docker containers. Furthermore, in order to use Kafka, another Apache service needed to be installed. Its name is Apache ZooKeeper and it is an open-source server used to coordinate distributed cloud applications. It basically offers a hierarchical key-value store that can be used in different ways and for different reasons. Given that we are generating a stream from a static dataset, we use ZooKeeper to store the current offset so the producer could know where to start again in case it crashes.Finally, we need to use a schema registry in order to enable the serialization of messages. Apache’s data serialization service is named Apache Avro. Every serialization schema that is defined and used in the producer is automatically added to the schema registry. One of the best things about Avro is that it can recognize which schema from the registry was used to generate a certain message on its own, without the need to specify the schema in the consumer. Kafka setup First, we need to launch an EC2 instance. Details about the instance we had chosen have already been discussed in the Infrastructure section. The second step is to download and install Docker which can be done in two ways: Docker Desktop (Docker with UI client) – official instructions Docker without UI client – instructions vary depending on your OS After that, we can install Kafka via Bitnami Kafka Docker image. There is a docker-compose.yml file that contains ZooKeeper dependency. However, that file sets up Kafka with a single broker and without the ability to connect external producers and consumers to the broker. External producers and consumers are those outside of a docker container and outside the EC2 instance. In other words, if the machine on which a producer or a consumer is being run has a different IP address from the EC2 instance, they will not be able to connect to the broker. You would probably want to have producers and consumers hosted on the EC2 instances inside the same VPC like Kafka, but when you are developing and testing it is convenient to start the producer or consumer from your local machine. That means we have to make some changes inside the docker-compose.yml file in order to make Kafka work the way we want. The most important ones are adding a service entry for each broker and editing the KAFKA_CFG_ADVERTISED_LISTENERS variable. This is crucial because of the way connections are handled inside Kafka. When a client starts, it connects to a broker and requests information about the partition leader broker. That happens because a client needs to connect to the partition leader broker in order to write or read messages. By default, the address that the client gets is a localhost address. Because of that, the client that is external might not be able to reach the partition leader broker. That’s why it’s important to add information about the public IP address and a reachable port. Also, we need to configure the schema-registry in docker-compose.yml file which is shown below.After taking care of everything aforementioned, this is what the final version of our docker-compose.yml file looks like: Implementation details Python library All of our code is written in Python. At first, we had planned to use the kafka-python library, but later we switched to confluent_kafka. The reason for this change was the fact that the kafka-python library doesn’t support asynchronous message sending which means we wouldn’t be able to define callback functions. Streaming Given that we are trying to stream static data, we had to somehow simulate a data source. We did that by creating a data generator that sequentially creates messages that are eventually loaded and processed by the producer. The interesting thing is that in the original dataset one entry represents sales data for one item across all available days (as sales data is stored in
Let’s dive into data science tools and algorithms
Croatia Osiguranje & BIRD Incubator Data Challenge – Part 2 After we explained fundamental data science concepts and techniques in the first part of this post, the second one will be about the tools and algorithms that were used during this Data Challenge. Data science tools As always, it is very important to use the right tool for the right job when working on a data science project. Additionally, it is important to follow the trends and to be familiar with the tools and technologies available at the moment. We choose our technologies carefully, keeping in mind the standards we have to fulfill to satisfy ourselves and our clients. This section gives an insight into the technologies we, as well as millions of others, use every day trying to make our code more efficient and the user interface more inviting. Python data science stack We used data science libraries that are widely used in the industry when working in Python: scikit-learn, pandas, seaborn. Scikit-learn is a machine learning library that is simple, easy to use, and very efficient. Pandas is a fast and flexible data analysis and data manipulation software library. Seaborn is a statistical data visualization library based on matplotlib that provides a high-level interface for drawing informative and interesting statistical graphics. Also, we used Jupyter notebook, a commonly used open-source web application intended for creating and sharing documents with blocks of live code and visualizations. Furthermore, for forecasting, we used pmdarima, a statistical library intended to be used when working with time series, and for stationarity checks, we used statsmodels, a statistical library for statistical tests and models estimation. We had to adjust some parts of the code to our problem, but we didn’t write any machine learning algorithms from scratch. Anvil Anvil is a Python platform for developing easy, modern, and powerful web applications. It allows the programmer to design an interface using the drag and drop technique. It integrates with Jupyter notebooks and can be easily deployed. Its idea is to free programmers of repetitive tasks and allow them to design attractive and useful applications easily and without complications. Also, it provides non-technical staff with the opportunity to use machine learning without having to go through a ton of tutorials and educational materials. Segmentation algorithms Clustering is a complex problem in the domain of unsupervised learning. Deciding which models to use can sometimes be a very difficult problem which is why we decided to keep it simple. Analyzing the nature of data and the nature of available clustering models resulted in us picking only two models, one deterministic and the other probabilistic. The probabilistic one, the Gaussian mixture, turned out to be our choice for this Data Challenge. The biggest challenge was to find the optimal number of clusters. Ultimately, we did it by comparing an existing variable in the dataset with the results of multiple variations of our models, each initialized with a different value for the number of clusters. To be exact, it was done by comparing the distribution of a variable that represents the segmentation according to the actual business rules and the distribution of clusters generated by our models. Also, it’s important to point out our idea wasn’t to perfectly fit these two distributions – that would turn this problem into a supervised learning problem. Our idea was for the existing variable to simply steer us in the right direction, but still detect new rules and come to a new conclusion. K-Means An iterative algorithm with the main goal of partitioning the dataset into K clusters where each observation belongs to only one cluster, the one with the nearest mean. It is a simple, general-purpose algorithm that minimizes the distance between points and the cluster’s centroids. Gaussian mixture A Gaussian mixture model is a probabilistic model that assumes all data points come from a mixture of a finite number of Gaussian distributions with unknown parameters. In short, the algorithm first chooses random cluster parameters and then tunes them until the change in parameters is less than the predefined threshold or the maximum number of iterations has been reached.Despite the Gaussian mixture algorithm being a probabilistic one and the K-Means algorithm being a deterministic one, there are some similarities among them; e.g. initial parameter for both models is the final number of clusters. In the end, Gaussian mixture models gave much better results during this Data Challenge which is why the final solution is generated using GMM. Forecasting algorithms The most natural solution to insurance market forecasting seemed to be creating a time series and using an ARIMA model. There are multiple reasons for us taking this approach, the main ones being the lack of details in the data and the seasonal nature of the data. It turned out that ARIMA gave quality forecasts we were satisfied with and we stuck with it until the end of the project, regardless of the fact we tried VAR, the more complicated model similar to ARIMA that didn’t provide us with better results. ARIMA ARIMA was the name that instantly popped into our heads when we read the task description for this Data Challenge. ARIMA stands for Autoregressive integrated moving average, it is a statistical model that uses past values of the time series to predict the future points in the time series. If it’s a non-seasonal ARIMA, it has 3 parameters: order of the autoregressive model, degree of data differencing, and the order of the moving-average model. In the case of a seasonal ARIMA, there are 6 parameters, each of the above-mentioned parameters for non-seasonal and seasonal parts of the model. Let’s look at the comparison of two ARIMA models, one that did poorly and the other one that produced good results. The first model is the one with poor performance. We can see that by looking at all metrics – r-squared is much lower and all errors are much higher. Perhaps the most intuitive metric to look at is MAPE (Mean Absolute Percentage
Let’s dive into time series forecasting concepts
Croatia osiguranje & BIRD Incubator Data Challenge – Part 1 Data science isn’t only about using fancy visualization tools and robust machine learning algorithms. There is a lot of manual hard work beneath the shiny and aesthetically pleasing solutions. In this blog post, divided into two parts, we tried to sum up data science concepts, technologies, and algorithms we used while working on Croatia osiguranje & BIRD Incubator Data Challenge. Let’s dive into the first part, the one where we will explain basic data science concepts and techniques that were necessary to complete this Data Challenge. Data science techniques Every data scientist knows that before letting the machine do its job and learn something from the data, data needs to be preprocessed using a number of concepts and techniques that play a key role in ensuring the quality of the final model. Also, after the machine does its job and produces the model we asked for, we should check if the model is good enough for us. In most situations, we will want to have a large number of models that we will later compare using model evaluation techniques. This section gives a brief overview of key data science concepts and techniques that were used during this Data Challenge to ensure the final solution is the best one possible. Data cleansing One of the first steps in any data science project should be removing rouge data. Corrupt, inaccurate, and inconsistent data can lead to incorrect conclusions which means detection and removal of these records from the dataset are crucial for the project’s success. We were on the lookout for non-existent data, duplicates, and outliers. It’s important to note our dataset is large enough to make it safe to remove the rouge data completely. Had it been significantly smaller, we would have to fill the missing values or replace the ones we detected as problematic. The existence of rouge data in this dataset can be accredited to the fact that every data entry was entered manually. This is something that should be avoided whenever possible. Feature scaling Also, an important part of data preprocessing, feature scaling, is performed to ensure all values fit into a predefined range. This is important because data distribution can highly affect the quality of machine learning model outputs. To know which feature scaling method should be used and if it should be used at all, one should be familiar with the nature of a machine learning model that is being used and the data itself. After preliminary analysis, we noticed there was a significant difference between ranges of features in our dataset. That often happens when working with a dataset formed with different types of features. In this case, there were a lot of financial features of different nature, as well as features regarding the number of employees.The most frequently used feature scaling methods are Min-Max Normalization, which we used for the course of this project, Mean Normalization and Standardization, also known as Z-Score Normalization. Stationarity When working with time series, a series of data points indexed in time order, stationarity is a very important concept one should keep in mind at all times. A stationary time series is the one whose mean and variance do not change over time. Differently put, a stationary time series is the one whose statistical properties do not depend on the moment at which the series is observed. If these properties vary over time, that kind of time series is called a non-stationary time series. If a time series is non-stationary, it means it follows one of three possible behavioral patterns: trend, seasonal or cyclic. In each of these cases, we have to transform the time series into a stationary time series. The most commonly used technique for such a task is called differencing, explained in the next subsection. Detecting stationarity can be done using various methods: looking at the ACF plots, analyzing summary statistics of multiple randomly selected data periods, and conducting statistical tests, the most popular ones being the Augmented Dickey-Fuller test and Kwiatkowski-Phillips-Schmidt-Shin test.We decided to use statistical tests, both ADF and KPSS. Given that we used ARIMA models, differencing was made automatically as part of the forecasting process. In most of the cases, ARIMA managed to achieve stationarity, but some time series remained non-stationary even after differencing. Differencing Differencing is a technique used in data science projects in order to stabilize the mean and variance of a time series over time. This is made by computing differences of consecutive terms in the time series. Equation 1: First difference of a time series In most of the cases, a first order difference is enough to make the time series stationary. However, sometimes a difference of higher order must be used. Equation 2: General expression for n-order difference of a time series The differencing technique should be applied multiple times until the time series becomes stationary according to the stationarity check techniques. An example of ACF plot before and after differencing is shown on figures 1 and 2. On the first figure a seasonal pattern can be noticed, while the differences are more randomly dispersed on the second figure. Figure 1: ACF plot before differencing Figure 2: ACF plot after first-order differencing As already mentioned, ARIMA automatically conducts differencing as part of its forecasting process, so we didn’t have to worry about differencing. If we hadn’t worked with ARIMA, we would have to manually do the job. The trouble with manual differencing is that predictions produced with differenced time series also represent differences. That means we have to obtain an inverse difference of the model’s output which makes the implementation more complicated and error-prone. Also, in the process of differencing and inverse differencing some entries are inevitably lost. That can be especially important when working with smaller datasets. Model evaluation Techniques used for model evaluation differ depending on the problem that is being solved. In case of unsupervised machine learning models, such as clustering, there isn’t a