Operations monitoring platform
Overview
Managing the production process of raw materials is a heavy task when it comes to monitoring data and interpreting results so you can make informed decisions. In any production cycle, you are dependent on data, either internal or external, to help guide you through the process. For that, a solution has to be adopted to gather and convert data into actionable insights.
In any production environment, you have machines and devices that automate the production processes. They all collect or generate data that track the cycle from beginning to end. Each piece of information is vital in describing the process and it could be used for reporting and decision making. In the process of creating the end product or output, some steps have to be followed and optimized. Using available data from the production process can be of great significance in achieving operational efficiency.
Our work in this solution was oriented toward data collection and establishing data quality that can be used in a platform that will provide analytics and reporting. First, we established data sources and what could be used that is important for users. We had to define which metrics are most useful and of significance in the production process. But, not all data came in the same format or of the same quality. It was integral to combine all relevant data into one repository and structure it the way it could be utilized for further analysis or data science processes.
By using internal data from accompanying systems in production, that collect or generate data, and external data, like macro or micro influences, we set off to create metrics and insights that could be used for production optimization. That information can be also used to spot errors, issues, or discrepancies which in turn helps users lower costs or avoid losses.
From showing the current production status to reporting, this solution is a basis to include various departments, organizational units, or people of interest in decision making. It is a platform made to monitor daily operations and present results. What is optimal in this platform is that through visualization it presents data for easier interpretation and sharing. Users can more easily see trends and historical data as opposed to looking through endless spreadsheets.
Project scope
Discovery:
- Client interviews
- As is system analysis
- Data validation
- Project planning
Procedure:
- Data collection & consolidation
- ETL pipelining
- Data warehousing
- Data analytics
Team:
- Data scientists
- Data engineers
Project duration:
- 2 years in terms of data engineering
- Ongoing: maintenance, and some new feature implementations
Project goal:
- Collect, combine, and present data from multiple sources in order to provide insights into the business processes and track performance
Methods:
ETL pipelines:
- We have implemented and improved ETL pipelines for data extraction, processing, and cleaning
KPI calculations:
- Data was stored in a data warehouse where business logic was implemented and all-important KPIs and metrics were calculated by combining all data sources
Results:
- Finally, data is replicated to data marts which serve data to the application users
Technologies used:
Vertica
PostgreSQL
Apache Ignite
Pentaho Data Integration ETL tool
Python
CRON
Apache Airflow
Apache Spark
AWS Cloud
Bash
Corn
What we wanted to accomplish:
Main issues:
What we needed to accomplish:
- Collect and structure data from multiple sources
- Find valuable data that will provide the best results
- Turn data into actionable metrics and insights
- Save users time on data collection and interpretation
- Minimize errors
Main issues:
- Encountering bad data
- Consolidation of multiple data sources with different types of data
- Deciding on which sources provide the most relevant data and which don’t
- How to combine internal and external sources of data to provide unified metrics
Our process:
A data analytics part of the solution is created to convert data into meaningful and actionable insights and metrics. The process consists of data collection, transformation, and analysis. Not all data comes in the same formats and volume. There have to be systems built to draw data from their sources into one place and in formats that can be used to present information relevant for making decisions on daily business operations. Our approach is to find valuable data that will be analyzed and presented to users.
The goal of data analytics is to convert data into numbers that users can easily understand and interpret. In line with that, they can easily make assumptions and analyze information. Through data engineering and data science methods, data is extracted and put through scientific methods, algorithms, and processes so the software solution can communicate findings on business operations, especially production.
Solution:
As a part of a web software solution that Deegloo worked on, it was necessary to also develop methods to collect data from multiple sources in milk production and farm management. Great amounts of data are generated daily and there was a need for data aggregation and visualization so users can receive insights that are a basis for making informed strategic decisions.
Farms collect data on milk, herd, and feed, and that information needed to be compared to weather information, milk quality, and price data. Data like that comes from internal and external sources in different formats, so we used multiple methods to be able to aggregate that data and turn it into easy-to-interpret results.
Results:
Data analysis and KPI calculation
Production insights and metrics all in one place
Multiple data sources were connected in one cohesive context
Easy visualization of daily generated data and trends
Admired by
