Responsibilities:
Create and maintain optimal data pipeline architectures
Collaborate with the team to decide on which tools and strategies to use within specific data preparation scenarios
Work with stakeholders to assist with data-related technical issues and support their data infrastructure needs
Monitor and anticipate trends in data engineering, and propose changes in alignment with organizational goals and needs
Share knowledge with other teams on various data engineering or project related topics
Required experience and skills:
At least 5 or more years of experience working in the data engineering field
Experience in building and optimization of high volume batch and real-time ETL data pipelines
Experience with Azure (Azure Data Lake Gen 2, Azure Data Factory, Azure Stream Analytics, Azure Databricks)
Extensive experience with Apache Flink (Flink SQL, PyFlink)
Experience with Apache Kafka
Experience with workflow orchestration tools (AirFlow)
Strong knowledge of Python language and SQL
English – Upper-Intermediate
Would be a plus:
Experience with ML
Experience with Java, Scala
Experience with Docker
Experience with Debezium CDC
Experience with Confluent Schema Registry
Experience with Hadoop ecosystem
Experience with setting up from scratch Big Data processing systems
Experience in AGILE methodologies
Cloud certifications
Knowledge of automation services (Terraform, Azure DevOps, AWS CodeBuild, Jenkins)
Knowledge of visualization technologies (Microsoft PowerBI)
Data modelling skills