We are seeking a highly skilled Platform Engineer with expertise in Data Engineering. As a Platform Engineer, you will be instrumental in designing, implementing, and maintaining our data streaming and processing infrastructure. Your role will involve collaborating with cross-functional teams to build efficient, scalable, and reliable solutions that enable real-time data integration, analytics, and insights.
Key Responsibilities
Collaborate with Data Engineers and Data Scientists to understand data requirements and create pipelines for real-time data ingestion, transformation, and processing.
Collaborate with DevOps teams to automate deployment, monitoring, and scaling of Kafka clusters and associated services.
Responsible for building and monitoring the Data platforms to ensure stability and maintain the disaster recovery capabilities.
Implement end-to-end data processing workflows using Kafka Streams, Azure Databricks, or similar technologies.
Monitor Kafka cluster health, troubleshoot and resolve issues related to performance, availability, and data consistency.
Qualifications:
Production Kubernetes Experience
Experience with Azure/MS Data Platforms: Postgres Single and Flex, Azure SQL, MS SQL server
Terraform (Preferred), Bicep or other IaC.
Microservice deployments using Helm (helm deployments as well writing charts).
Ability to create, maintain, and troubleshoot Azure DevOps pipelines.
Proven experience as a Platform Engineer, Data Engineer, or similar roles with a strong focus on Kafka and data engineering within the Azure environment.
Experience with Data pipelines for deploying database code – Postgres, Azure SQL, and SQL Server
Strong communication skills to collaborate effectively with cross-functional teams and stakeholders.
Data Movement, ADF, SSIS, Power BI, MS SQL, PostgreSQL, Kafka