Sr. Databricks Engineer

Posted on 19 October 26 by Chloe Schumaker

$ - $ Job Description

Capstone IT is helping our client to find a Senior Data Architect - Databricks who will lead the design, implementation, and modernization of enterprise data platforms using Databricks and cloud technologies.

This individual will be responsible for building scalable, secure, and high-performing data environments from initial architecture and configuration through production deployment. They will design Databricks Lakehouse solutions, develop data pipelines, implement data governance, and help migrate legacy data systems into modern cloud platforms.

The ideal candidate combines strategic architecture experience with strong hands-on engineering skills. They must be comfortable making technical decisions, writing code, configuring environments, troubleshooting complex issues, and leading engineering teams.

The biggest differentiator: Our client needs someone who has personally led a complete Databricks implementation—not just someone who has worked within an established Databricks environment.

Top 4 Required Skills

1. Databricks Architecture & End-to-End Implementation

3+ years of hands-on Databricks experience, including at least one enterprise implementation from initial architecture and environment setup through production.

2. Python, PySpark, Apache Spark & SQL

Advanced data engineering skills, including building and optimizing distributed data pipelines and ETLELT solutions.

3. Cloud Data Architecture & Lakehouse Design

Strong Azure, AWS, or GCP experience with Delta Lake, medallion architecture, cloud storage, and enterprise platform modernization.

4. Unity Catalog, Governance & Technical Leadership

Experience implementing governance, security, access controls, data quality, architectural standards, and guiding engineering teams.

Main Responsibilities

Architect and deploy enterprise Databricks Lakehouse platforms from the ground up.

Lead environment configuration, data integration, migration, testing, and production deployment.

Design data pipelines using Python, PySpark, SQL, Apache Spark, and Delta Lake.

Implement Bronze, Silver, and Gold data layers.

Establish governance and security using Unity Catalog.

Modernize legacy data warehouses and ETL platforms.

Optimize compute performance, scalability, reliability, and cloud costs.

Implement CICD pipelines and infrastructure automation.

Develop architecture diagrams, standards, and technical roadmaps.

Provide hands-on technical leadership to engineering, DevOps, analytics, and BI teams.

Required Qualifications

Qualification

Requirement

Overall experience

8+ years in data architecture, engineering, or warehousing

Databricks

3+ years hands-on

Implementation

At least one complete Databricks implementation

Programming

Advanced Python, PySpark, and SQL

Cloud

Azure, AWS, or GCP

Lakehouse

Delta Lake, Unity Catalog, Spark

Data engineering

ETLELT, integration, modeling, optimization

Modernization

Legacy-to-cloud migration experience

Leadership

Architecture ownership, technical mentoring, stakeholder communication

Preferred:

Databricks certifications, Terraform, CICD, Azure Data Factory, AWS Glue, dbt, Airflow, Power BI, Tableau, streaming, and AIML workloads.

What Success Looks Like

The successful candidate delivers a production-ready Databricks platform, establishes secure and reusable architecture patterns, improves data processing performance, and enables engineering teams to build reliable analytics solutions independently.