Job Description

Urgent Requirement - Azure Databricks Expert

Join our team as a

Databricks Champion

with mixed data & infrastructure skills. Lead the architecture and optimization of enterprise-scale Azure Databricks platforms for a dynamic organization.

The Role

What You'll Do:

Architect Databricks Solutions

- Design and deploy secure workspaces with VNet injection and private endpoints

Implement Unity Catalog

- Build comprehensive data governance with fine-grained access controls

Optimize Performance

- Monitor, tune, and troubleshoot complex Databricks environments at scale

Ensure Security

- Integrate Azure Entra ID, SCIM provisioning, and Key Vault secret management

Drive Integration

- Design robust patterns between ADF and Databricks for enterprise data pipelines

Key Responsibilities

Platform Management:

Azure Databricks workspace design, deployment, and optimization Cluster policies, auto-scaling, and cost optimization strategies Performance tuning for Spark configurations and Delta Lake Expert-level troubleshooting for complex platform issues

Data Governance:

Unity Catalog metastore architecture and implementation Row/column level security with fine-grained access controls Data lineage tracking and audit compliance Managed tables and external volumes governance

Security & Access:

Azure Entra ID SSO integration and user synchronization SCIM provisioning for automated identity lifecycle Azure Key Vault integration for secure secret management ACLs implementation across ADLS Gen2 and Unity Catalog

Integration & Operations:

ADF-Databricks integration patterns and CI/CD pipelines Comprehensive monitoring with Azure Monitor and Log Analytics Disaster recovery and business continuity planning Platform documentation and best practices development

Essential Skills

Azure Expertise:

Azure Databricks

- Workspaces, clusters, Unity Catalog, Delta Lake

Azure Data Factory

- Integration patterns, pipeline orchestration

Azure Storage

- ADLS Gen2, hierarchical namespaces, lifecycle policies

Azure Security

- Entra ID, Key Vault, private endpoints, NSGs

Azure Networking

- VNets, peering, secure connectivity

Technical Skills:

Programming

- Python, Scala, SQL, PowerShell/Azure CLI

Spark Architecture

- Performance tuning, configuration optimization

DevOps

- CI/CD pipelines, Azure DevOps, GitHub Actions

Security

- RBAC, SCIM, data encryption, network security

Monitoring

- Azure Monitor, Log Analytics, alerting strategies

What We're Looking For

Experience:

5+ years

Azure cloud platform experience

3+ years

hands-on Azure Databricks experience

Proven track record

with Unity Catalog implementations

Enterprise-scale

platform architecture and optimization

Strong problem-solving

skills for complex multi-service issues

Preferred:

Microsoft Certified: Azure Solutions Architect Expert

Databricks Certified Data Engineer Professional/Associate

Experience

with large-scale data platforms and governance

Why This Role?

Technical Leadership

- Drive architecture decisions for enterprise Databricks platform

Cutting-Edge Technology

- Work with latest Azure services and Databricks features

Business Impact

- Enable data-driven decision making across the organization

Professional Growth

- Expand expertise in cloud-native data platforms *

Collaborative Environment

- Work with data engineers, scientists, and security teams