Senior Databricks Engineer
Bengaluru
API International Capability Centre Private Limited
Key Responsibilities:
Databricks Platform Engineering
•
Lakehouse Architecture: Design and implement scalable Lakehouse architectures using Databricks, Delta Lake, and Unity Catalog as the primary platform.
•
Delta Live Tables: Build and manage DLT pipelines for declarative, reliable, and maintainable data transformation workflows.
•
Unity Catalog: Govern data assets — tables, volumes, and models — through Unity Catalog, enforcing fine-grained access control and lineage tracking.
•
Performance Optimization: Tune Spark jobs and Delta tables using Z-ordering, liquid clustering, auto-optimize, and partition strategies to maximize query performance.
•
Databricks Workflows: Orchestrate multi-task pipelines using Databricks Workflows, including job scheduling, dependency management, and alerting.
Data Pipeline & Integration
•
ELT/ETL Pipelines: Design, develop, and maintain robust ELT/ETL pipelines ingesting data from APIs, databases, streaming sources, and flat files into the Lakehouse.
•
Azure Integration: Integrate Databricks with Azure Data Factory, ADLS Gen2, Azure Synapse, Event Hubs, and other Azure-native services.
•
Streaming: Build real-time data processing pipelines using Structured Streaming and Delta Live Tables for low-latency analytics use cases.
Software Engineering & DevOps
•
Code Quality: Write clean, efficient, and reusable PySpark, Python, and SQL code following SOLID principles and team coding standards.
•
CI/CD: Implement and manage CI/CD pipelines in Azure DevOps for automated testing, deployment, and release of Databricks notebooks and jobs.
•
Infrastructure as Code: Provision and manage Databricks workspaces, clusters, and supporting Azure infrastructure using Terraform or Bicep.
•
Databricks Asset Bundles: Package and deploy Databricks projects using DABs for reproducible, environment-consistent deployments.
Data Governance & Quality
•
Data Quality Frameworks: Implement automated data validation, expectation checks, and quality monitoring using Great Expectations or DLT expectations.
•
Data Lineage: Maintain end-to-end data lineage visibility through Unity Catalog and Microsoft Purview integration.
•
Documentation: Produce and maintain thorough technical documentation for pipelines, data models, runbooks, and architectural decisions.
Collaboration & Leadership
•
Mentorship: Coach and mentor junior and mid-level engineers, conducting code reviews and sharing best practices.
•
Stakeholder Engagement: Translate business requirements into robust technical solutions; communicate trade-offs clearly to non-technical stakeholders.
•
Standards & Governance: Contribute to and enforce team engineering standards, SDLC processes, and platform governance policies.
Required Qualifications
•
Databricks Certified Data Engineer Professional or Associate (mandatory).
•
Microsoft Certified: Azure Data Engineer Associate (mandatory).
•
10+ years of hands-on experience in Data Engineering roles.
•
Deep expertise in Databricks — Delta Lake, Unity Catalog, Delta Live Tables, Workflows, Clusters.
•
Advanced PySpark and Python skills for large-scale distributed data processing.
•
Strong SQL skills: query authoring, performance tuning, and data modeling.
•
Experience with Azure Data Factory, ADLS Gen2, Azure Synapse Analytics.
•
Hands-on experience with Azure DevOps CI/CD and Infrastructure as Code (Terraform or Bicep).
•
Strong understanding of Data Warehousing concepts, dimensional modeling, and Data Lakehouse patterns (Medallion architecture).
•
Experience with Structured Streaming or other real-time data processing frameworks. Preferred Qualifications
•
Experience with Microsoft Purview for data cataloguing and governance.
•
Familiarity with dbt (data build tool) for SQL-based transformation workflows.
•
Knowledge of MLflow and Databricks Feature Store for ML engineering integration.
•
Exposure to Databricks Asset Bundles (DABs) for project packaging and deployment.
•
Experience with Azure Function Apps, Logic Apps, or Event Hubs for event-driven architectures.
•
Background in data mesh or data product thinking. Key Skills at a Glance Core Platform Databricks, Delta Lake, Unity Catalog, DLT Languages PySpark, Python, SQL
Cloud & Integration
Azure Data Factory, ADLS Gen2, Synapse, Event Hubs
DevOps & IaC
Azure DevOps, Terraform, Bicep, DABs
Data Quality & Governance Great Expectations, DLT Expectations, Purview Architecture Lakehouse, Medallion Architecture, Data Mesh Why Join Us?
•
Lead the build-out of a modern, Databricks-first data platform at scale.
•
Work with a collaborative, forward-thinking Data & Analytics team.
•
Access to cutting-edge tooling across Azure, Databricks, and the broader modern data stack.
•
Competitive salary, benefits, and a strong commitment to your professional development and certifications.
•
Opportunity to shape team standards, architecture decisions, and engineering culture.
•
Contribute to high-impact, global data initiatives.
Featured job listings
Each listing including the job title, location, brief
description, and a link to a page with more details.
