Design, build, and optimize scalable ETL data pipelines using Apache Spark, Kafka, and Airflow. Develop high-performance batch and streaming data workflows, ensure data quality and security, and collaborate with architects to deliver reliable data solutions.
Design and implement end-to-end ETL data pipelines using Apache Spark (batch and streaming) with Java.
Create, schedule, and monitor complex data workflows using Apache Airflow DAGs to ensure job execution and dependency management.
Deep understanding of Kafka architecture, including brokers, partitions, topics, including producers, consumers, and connectors.
Perform performance tuning on Spark jobs and Kafka clusters to maximize throughput, reduce latency, and ensure scalability.
Strong SQL skills and experience with NoSQL databases.
Ensure data quality, consistency, and security across the entire data lifecycle.
Work with data architects to translate business requirements into robust technical solutions.
Apache Spark, Java, ETL, Apache Airflow, DAGs, Apache Kafka, Kafka Producers, Kafka Consumers, Kafka Connect, Batch Processing, Stream Processing, SQL, NoSQL, Data Pipelines, Data Integration, Performance Tuning, Workflow Scheduling, Data Quality, Data Security, Data Modeling, Troubleshooting, Scalability, Linux, Git