Mid/Senior Data Engineer (hybrid)
The platform is used by US milk producers for easier production management and reporting through interactive dashboards. It is one of the solutions in an ecosystem that supports over 15,000 farmers that produce 100+ billion pounds of milk.
The operations monitoring platform collects data from various sources (S3 buckets, SFTP servers, APIs…) into a powerful data warehouse (Vertica database) that is optimized for data crunching. Data is then prepared with data cleaning and validation processes, after which the business logic is applied in the form of SQL scripts that combine data and calculate various KPIs. Results are stored in the data warehouse and replicated to a separate data mart database. The whole process is implemented as a sequence of ETL jobs that are orchestrated and monitored.
Typical tasks include maintenance of ETL jobs (changing the business logic), data warehouse (changing definitions of tables and views), and data mart (changing definitions of tables and views). Apart from maintenance, you can expect the implementation of new ETL jobs and modeling new tables in the data warehouse and data mart.
This solution blends data from multiple solutions from the aforementioned ecosystem and enables users to build their own reports and dashboards (self-servicing) and to monitor business processes through pre-build reports and dashboards.
Typical tasks include dataset preparation — creating tables in the database and implementing business logic that combines data from the data warehouse into new KPIs. The interesting part is that the prepared data is not just displayed to a user, but prepared in a way that a user can filter or combine data by filters and parameters, or even build their own dashboards and reports. Advanced row-level security that enables automated authorization is another interesting aspect of this solution.
We have ensured a smooth onboarding process with a senior developer that is working on projects for more than 3 years. After the onboarding, you’ll gradually take on more and more responsibilities.
© 2022 DigitalPoirots.com | Deegloo.com
Your message was sent successfully!