Senior Databricks & Azure Platform Admin

Overview

The Senior Databricks & Azure Platform Administrator is responsible for administering, optimizing, and securing enterprise Databricks environments on Azure while ensuring high platform availability, performance, and scalability. The role involves managing Databricks workspaces and clusters, implementing CI/CD pipelines, configuring advanced Databricks features, driving FinOps initiatives, automating administrative tasks, supporting data engineering teams, and ensuring compliance with cloud security and governance standards. The ideal candidate has extensive experience with Databricks administration, Apache Spark, Azure cloud services, scripting, DevOps, and enterprise data platforms.

Job Description

Key Responsibilities

• Administration and Management: Administer and manage the Enterprise Databricks platform, including user access, cluster management, workspace management and resource allocation.
  Monitor system performance, identify and resolve issues, and ensure high availability and reliability of the Databricks environment.
• CICD pipelines build - For release management, integration with git or similar code base.

• Databricks Setup - Setup and configure Databricks features like Lakebase, Lakeflow, Feature engineering, Genie space, AI/ BI and others 

• FinOps - Implement FinOps to reduce platform cost and enable chargeback to respective cost centres

• Configuration and Optimization: Configure Databricks clusters and workspaces to meet the specific needs of data processing and analytics workflows.
  Optimize cluster performance and resource utilization to improve efficiency
• Security and Compliance: Implement and maintain security policies and access controls to protect sensitive data.
  Ensure compliance with data governance and regulatory requirements.
• Support and Troubleshooting: Provide technical support to users, troubleshoot issues, and resolve problems related to Databricks.
  Collaborate with data engineers and data scientists to understand their needs and provide solutions.
• Automation and Scripting: Develop scripts and automation tools to streamline administrative tasks and improve efficiency.
  Automate the deployment and scaling of Databricks clusters and workspaces.
• Documentation and Training: Create and maintain comprehensive documentation for Databricks configurations, processes, and procedures.
  Conduct training sessions and workshops to educate users on best practices for using Databricks.
  
  
  
• Proven experience as a Databricks Admin, Systems Administrator, or similar role. 
• Hands-on experience with Databricks administration and Apache Spark. 
• Experience with cloud platforms such as AWS, Azure, or Google Cloud Platform.
• Proficiency in scripting languages such as Python, Bash, or PowerShell. 
• Strong understanding of data processing and ETL workflows. 
• Familiarity with networking, security, and identity management in cloud environments.
• Certifications in Databricks, Apache Spark, or related technologies.
• Experience with DevOps practices and tools for continuous integration and deployment.
• Knowledge of data governance and compliance standards.
• Excellent problem-solving and analytical skills. Strong communication and collaboration abilities. Ability to work effectively in a fast-paced, dynamic environment.

 

Skills & Requirements

Apply Now