Databricks Program Director

Job Summary The Databricks Program Director will lead the modernization and migration of a Databricks Lakehouse Platform, overseeing the design of scalable architectures, development of automated data pipelines, and implementation of enterprise data solutions. This role involves architecting multi-layer Lakehouse models, collaborating with business stakeholders, partnering with data engineering and data science teams, optimizing performance, ensuring strong governance, and leveraging AI copilots to accelerate development. The Program Director will work across functional teams to deliver high-impact, cloud-native data platforms that support advanced analytics and strategic decision-making. Key Responsibilities Design & Architecture of Scalable Data Platforms Design, develop, and maintain large-scale data architectures on the Databricks Lakehouse Platform. Architect multi-layer Lakehouse models (Bronze, Silver, Gold) for various business domains. Utilize Delta Lake, Unity Catalog, and Databricks platform features for??, versioning, and reproducibility. Client & Business Stakeholder Engagement Translate functional requirements into scalable technical solutions. Lead architecture workshops and solutioning sessions with business and IT stakeholders. Data Pipeline Development & Collaboration Collaborate with engineering and data science teams to build end-to-end pipelines using PySpark, SQL, DLT, and Databricks Workflows. Enable ingestion from multiple data sources including ERP, POS, CRM, ecommerce, and third-party datasets. Performance, Scalability & Reliability Optimize Spark jobs for performance, scalability, and cost efficiency. Implement monitoring using Databricks Observability, Ganglia, and cloud-native tools. Security, Compliance & Governance Design secure architectures using Unity Catalog, RBAC, encryption, and lineage tools. Establish governance practices such as Data Fitness Index, Quality Scores, SLA monitoring, and metadata cataloging. AI Copilots & Agentic Development Use AI tools such as GitHub Copilot and Databricks Assistant for code generation, documentation, debugging, and optimization. Advocate for agentic workflows supporting data profiling, schema inference, automated testing, and validation. Innovation & Continuous Learning Stay current with trends in Lakehouse architecture, Generative AI, and cloud-native tools. Evaluate new Databricks features and partner integrations for platform improvements. Required Qualifications Bachelor's or Master's degree in Computer Science, Information Technology, or a related field. 12-18 years of hands-on data engineering experience, including 5+ years in Databricks architecture and Apache Spark. Expertise in building high-throughput, low-latency ETL/ELT pipelines using PySpark, SQL, and Databricks-native features. Experience with data ingestion from APIs, files, RDBMS, and cloud storage (ADLS Gen2). Strong experience designing Lakehouse architectures with Bronze, Silver, and Gold layers. Deep knowledge of Delta Lake performance optimization (OPTIMIZE, VACUUM, ZORDER, Time Travel). Solid understanding of data modeling (star/snowflake, dimensional modeling). Experience designing Data Marts using Databricks SQL Warehouse and integrating with BI tools. Hands-on experience with Workflows, Delta Live Tables (DLT), Unity Catalog, and MLflow. Experience with Databricks REST APIs, Notebooks, and cluster configuration. Experience integrating Databricks into CI/CD pipelines with Azure DevOps or GitHub Actions. Knowledge of IaC using Terraform or ARM templates. Experience with Azure services such as ADF, Synapse, ADLS, Key Vault, and Azure Monitor. Strong knowledge of data privacy, access controls, and governance frameworks. Experience with Unity Catalog, RBAC, tokenization, and data classification. Excellent communication skills for stakeholder engagement, architecture presentations, and cross-team collaboration. Proven experience mentoring or leading global cross-functional teams. Ability to work independently in agile or hybrid delivery models. Preferred Qualifications Experience with Generative AI or agentic workflows for data engineering. Familiarity with MDM, enterprise metadata management, or data mesh practices. Experience in retail, CPG, manufacturing, or supply chain analytics.