Operations monitoring platform

Overview

Managing the production process of raw materials is a heavy task when it comes to monitoring data and interpreting results so you can make informed decisions. In any production cycle, you are dependent on data, either internal or external, to help guide you through the process. For that, a solution has to be adopted to gather and convert data into actionable insights.

In any production environment, you have machines and devices that automate the production processes. They all collect or generate data that track the cycle from beginning to end. Each piece of information is vital in describing the process and it could be used for reporting and decision making. In the process of creating the end product or output, some steps have to be followed and optimized. Using available data from the production process can be of great significance in achieving operational efficiency.

Our work in this solution was oriented toward data collection and establishing data quality that can be used in a platform that will provide analytics and reporting. First, we established data sources and what could be used that is important for users. We had to define which metrics are most useful and of significance in the production process. But, not all data came in the same format or of the same quality. It was integral to combine all relevant data into one repository and structure it the way it could be utilized for further analysis or data science processes.

 

By using internal data from accompanying systems in production, that collect or generate data, and external data, like macro or micro influences, we set off to create metrics and insights that could be used for production optimization. That information can be also used to spot errors, issues, or discrepancies which in turn helps users lower costs or avoid losses. 

From showing the current production status to reporting, this solution is a basis to include various departments, organizational units, or people of interest in decision making. It is a platform made to monitor daily operations and present results. What is optimal in this platform is that through visualization it presents data for easier interpretation and sharing. Users can more easily see trends and historical data as opposed to looking through endless spreadsheets.

Project scope

Discovery:

  • Client interviews
  • As is system analysis
  • Data validation
  • Project planning

Procedure:

  • Data collection & consolidation
  • ETL pipelining
  • Data warehousing
  • Data analytics

Team:

  • Data scientists
  • Data engineers

Project duration:

  • 2 years in terms of data engineering
  • Ongoing: maintenance, and some new feature implementations

Project goal:

  • Collect, combine, and present data from multiple sources in order to provide insights into the business processes and track performance

Methods:

ETL pipelines:

  • We have implemented and improved ETL pipelines for data extraction, processing, and cleaning

KPI calculations:

  • Data was stored in a data warehouse where business logic was implemented and all-important KPIs and metrics were calculated by combining all data sources

Results:

  • Finally, data is replicated to data marts which serve data to the application users

Technologies used:

Vertica

PostgreSQL

Apache Ignite

Pentaho Data Integration ETL tool

Python

Apache Spark logo

CRON

Apache Airflow

Apache Spark

AWS Cloud

Bash

Corn

What we wanted to accomplish:

Main issues:

What we needed to accomplish:

  • Collect and structure data from multiple sources
  • Find valuable data that will provide the best results
  • Turn data into actionable metrics and insights
  • Save users time on data collection and interpretation
  • Minimize errors

Main issues:

  • Encountering bad data
  • Consolidation of multiple data sources with different types of data
  • Deciding on which sources provide the most relevant data and which don’t
  • How to combine internal and external sources of data to provide unified metrics

Our process:

A data analytics part of the solution is created to convert data into meaningful and actionable insights and metrics. The process consists of data collection, transformation, and analysis. Not all data comes in the same formats and volume. There have to be systems built to draw data from their sources into one place and in formats that can be used to present information relevant for making decisions on daily business operations. Our approach is to find valuable data that will be analyzed and presented to users.

The goal of data analytics is to convert data into numbers that users can easily understand and interpret. In line with that, they can easily make assumptions and analyze information. Through data engineering and data science methods, data is extracted and put through scientific methods, algorithms, and processes so the software solution can communicate findings on business operations, especially production.

Solution:

As a part of a web software solution that Deegloo worked on, it was necessary to also develop methods to collect data from multiple sources in milk production and farm management. Great amounts of data are generated daily and there was a need for data aggregation and visualization so users can receive insights that are a basis for making informed strategic decisions. 

Farms collect data on milk, herd, and feed, and that information needed to be compared to weather information, milk quality, and price data. Data like that comes from internal and external sources in different formats, so we used multiple methods to be able to aggregate that data and turn it into easy-to-interpret results.

Results:

Data analysis and KPI calculation

Production insights and metrics all in one place

Multiple data sources were connected in one cohesive context

Easy visualization of daily generated data and trends

Admired by

Ryne Braun
Ryne BraunProduct Manager, Dairy.com
Read More
Part of the client’s success is attributed to Digital Poirots consistent, on-the-dot delivery. The team mitigates delays by proactively communicating with subject matter experts. They also provide thorough reports that eliminate the need for lengthy back-and-forths.
Marin Kosović
Marin KosovićLead Data Scientist, Bellabeat
Read More
Digital Poirots leads a precise execution, meeting the team’s requirements. They communicate effectively, establishing a seamless workflow. They were very prompt and precise in response. Their professionalism and expertise were impressive.
Previous
Next

Too much data?

Let's talk

Let us help you discover all the clues!

Croatia
Zadarska 80, Zagreb
HR62004514019
info@digitalpoirots.com

© 2023 DigitalPoirots.com | Deegloo.com