Data Engineer
Junior/Mid Data Engineer (hybrid)
As a data engineer, you will be working on interesting projects related to the operations monitoring platform and self-servicing visualization solution. You will collaborate with our internal team of 4 people, as well as with a client’s business and development team.
What you’ll be doing
The operations monitoring platform
One of your primary responsibilities will be working on the platform overseeing the majority of the US dairy industry. The platform is used by US milk producers for easier production management and reporting through interactive dashboards. It is one of the solutions in an ecosystem that supports over 15,000 farmers that produce 100+ billion pounds of milk.
The operations monitoring platform collects data from various sources (S3 buckets, SFTP servers, APIs, …) into a powerful data warehouse (Vertica database) that is optimized for data crunching. Data is then prepared with data cleaning and validation processes, after which the business logic is applied in the form of SQL scripts that combine data and calculate various KPIs. Results are stored in the data warehouse and replicated to a separate data mart database. The whole process is implemented as a sequence of ETL jobs that are orchestrated and monitored.
Typical tasks include maintenance of ETL jobs (changing the business logic), data warehouse (changing definitions of tables and views), and data mart (changing definitions of tables and views). Apart from maintenance, you can expect the implementation of new ETL jobs and modeling new tables in the data warehouse and data mart.
Self-servicing visualization solution
This solution blends data from multiple solutions from the aforementioned ecosystem and enables users to build their own reports and dashboards (self-servicing) and to monitor business processes through pre-build reports and dashboards.
Typical tasks include dataset preparation — creating tables in the database and implementing business logic that combines data from the data warehouse into new KPIs. The interesting part is that the prepared data is not just displayed to a user, but prepared in a way that a user can filter or combine data by filters and parameters, or even build their own dashboards and reports. Advanced row-level security that enables automated authorization is another interesting aspect of this solution.
Data pipeline migration (Optional)
If you are a mid-level engineer looking for an opportunity in driving process transformation, this role challenges you to apply a solid understanding of software engineering skills and DevOps experience.
In this role, you will potentially take part in the migration of over 100 data pipelines from traditional ETL architecture to a modern tech stack that separates compute and storage. You may help design and implement scalable data pipelines using cloud-native technologies to optimize data processing efficiency and cost-effectiveness. Tasks may involve supporting the evaluation and selection of tools and frameworks for data ingestion, transformation, and orchestration. Collaboration with cross-functional teams to ensure seamless integration and performance optimization, while maintaining data quality and reliability throughout the migration process, is emphasized. Additionally, you may participate in documenting workflows, providing training, and supporting stakeholders to facilitate the adoption of the new tech stack.
Onboarding process
We have ensured a smooth onboarding process with developers who are working on projects for more than 5 years. After the onboarding, you’ll gradually take on more and more responsibilities.
Qualifications:
- Technical requirements
- Minimum 6 months ETL / database / DWH developmnet experience
- Intermediate SQL knowledge
- Knowledge of ETL concepts
- Data collection
- Data cleaning processes
- Historical and incremental data synchronization
- Business logic implementation
- Job scheduling and monitoring
- Exception processing
- Knowledge of DWH concepts
- Database design (schema types)
- Staging tables, final tables
- Fact and dimension tables
- Data warehouse, data mart
- Experience with data integration tools (Pentaho / Boomi / Informatica / SSIS / ODI)
- Basic data visualization knowledge
- Knowledge of programming concepts
- Basic Python knowledge
- Basic Bash scripting knowledge
- Basic Docker knowledge
- Basic Linux administration knowledge
- Nice to have(s), especially if you are interested in data pipeline migration project:
- Software engineering skills
- DevOps skills (version control, containerization, CI/CD pipelines, Infrastructure as Code)
- Experience with Tableau or similar visualization tools
- Datasource preparation
- Building interactive dashboard
- Basic machine learning knowledge (regression analysis, clustering, cross-validation)
- Soft skills and personality traits
- Pleasant, easy to work with
- Professional, committed to work
- Adaptable, fast learner
- Proactive
- Good communication and presentation skills
Your responsibilities:
- Implementation of ETL pipelines, data warehouse, data mart, and related processes
- Implementation of Tableau reports and dashboards
- Monitoring and maintenance of Linux servers
- Everyday communication with team members in Croatia and USA
- Collecting technical requirements from the client through online meetings
We can offer you:
- A friendly and welcoming environment with good interpersonal communication
- Interesting and challenging project to work on with opportunity for advancement
- Flexible working hours, paid personal days, hybrid workplace
- Professional equipment in form of MacBook Pro and a 4k monitor
- Compensation package based on your level of experience and value provided
- An educational budget for any books you would like to read or certificates you would like to apply for
- Supplementary health insurance & health checks
- Multisport
- Birthday voucher, Christmas & Easter bonuses
- Performance bonus, monthly supplementary lunch bonus
