Key Responsibilities:
Databricks Platform Engineering
\u2022
Lakehouse Architecture: Design and implement scalable Lakehouse architectures using Databricks, Delta Lake, and Unity Catalog as the primary platform.
\u2022
Delta Live Tables: Build and manage DLT pipelines for declarative, reliable, and maintainable data transformation workflows.
\u2022
Unity Catalog: Govern data assets \u2014 tables, volumes, and models \u2014 through Unity Catalog, enforcing fine-grained access control and lineage tracking.
\u2022
Performance Optimization: Tune Spark jobs and Delta tables using Z-ordering, liquid clustering, auto-optimize, and partition strategies to maximize query performance.
\u2022
Databricks Workflows: Orchestrate multi-task pipelines using Databricks Workflows, including job scheduling, dependency management, and alerting.
Data Pipeline & Integration
\u2022
ELT\/ETL Pipelines: Design, develop, and maintain robust ELT\/ETL pipelines ingesting data from APIs, databases, streaming sources, and flat files into the Lakehouse.
\u2022
Azure Integration: Integrate Databricks with Azure Data Factory, ADLS Gen2, Azure Synapse, Event Hubs, and other Azure-native services.
\u2022
Streaming: Build real-time data processing pipelines using Structured Streaming and Delta Live Tables for low-latency analytics use cases.
Software Engineering & DevOps
\u2022
Code Quality: Write clean, efficient, and reusable PySpark, Python, and SQL code following SOLID principles and team coding standards.
\u2022
CI\/CD: Implement and manage CI\/CD pipelines in Azure DevOps for automated testing, deployment, and release of Databricks notebooks and jobs.
\u2022
Infrastructure as Code: Provision and manage Databricks workspaces, clusters, and supporting Azure infrastructure using Terraform or Bicep.
\u2022
Databricks Asset Bundles: Package and deploy Databricks projects using DABs for reproducible, environment-consistent deployments.
Data Governance & Quality
\u2022
Data Quality Frameworks: Implement automated data validation, expectation checks, and quality monitoring using Great Expectations or DLT expectations.
\u2022
Data Lineage: Maintain end-to-end data lineage visibility through Unity Catalog and Microsoft Purview integration.
\u2022
Documentation: Produce and maintain thorough technical documentation for pipelines, data models, runbooks, and architectural decisions.
Collaboration & Leadership
\u2022
Mentorship: Coach and mentor junior and mid-level engineers, conducting code reviews and sharing best practices.
\u2022
Stakeholder Engagement: Translate business requirements into robust technical solutions; communicate trade-offs clearly to non-technical stakeholders.
\u2022
Standards & Governance: Contribute to and enforce team engineering standards, SDLC processes, and platform governance policies.
Required Qualifications
\u2022
Databricks Certified Data Engineer Professional or Associate (mandatory).
\u2022
Microsoft Certified: Azure Data Engineer Associate (mandatory).
\u2022
10+ years of hands-on experience in Data Engineering roles.
\u2022
Deep expertise in Databricks \u2014 Delta Lake, Unity Catalog, Delta Live Tables, Workflows, Clusters.
\u2022
Advanced PySpark and Python skills for large-scale distributed data processing.
\u2022
Strong SQL skills: query authoring, performance tuning, and data modeling.
\u2022
Experience with Azure Data Factory, ADLS Gen2, Azure Synapse Analytics.
\u2022
Hands-on experience with Azure DevOps CI\/CD and Infrastructure as Code (Terraform or Bicep).
\u2022
Strong understanding of Data Warehousing concepts, dimensional modeling, and Data Lakehouse patterns (Medallion architecture).
\u2022
Experience with Structured Streaming or other real-time data processing frameworks. Preferred Qualifications
\u2022
Experience with Microsoft Purview for data cataloguing and governance.
\u2022
Familiarity with dbt (data build tool) for SQL-based transformation workflows.
\u2022
Knowledge of MLflow and Databricks Feature Store for ML engineering integration.
\u2022
Exposure to Databricks Asset Bundles (DABs) for project packaging and deployment.
\u2022
Experience with Azure Function Apps, Logic Apps, or Event Hubs for event-driven architectures.
\u2022
Background in data mesh or data product thinking. Key Skills at a Glance Core Platform Databricks, Delta Lake, Unity Catalog, DLT Languages PySpark, Python, SQL
Cloud & Integration
Azure Data Factory, ADLS Gen2, Synapse, Event Hubs
DevOps & IaC
Azure DevOps, Terraform, Bicep, DABs
Data Quality & Governance Great Expectations, DLT Expectations, Purview Architecture Lakehouse, Medallion Architecture, Data Mesh Why Join Us?
\u2022
Lead the build-out of a modern, Databricks-first data platform at scale.
\u2022
Work with a collaborative, forward-thinking Data & Analytics team.
\u2022
Access to cutting-edge tooling across Azure, Databricks, and the broader modern data stack.
\u2022
Competitive salary, benefits, and a strong commitment to your professional development and certifications.
\u2022
Opportunity to shape team standards, architecture decisions, and engineering culture.
\u2022
Contribute to high-impact, global data initiatives.<\/p>\n","protected":false},"featured_media":7156,"template":"","meta":{"_acf_changed":false,"inline_featured_image":false},"country":[17],"class_list":["post-7368","job_posting","type-job_posting","status-publish","has-post-thumbnail","hentry","country-india"],"acf":[],"yoast_head":"\n