Capstone IT is helping our client to find a Senior Data Engineer - Databricks who will lead the design, implementation, and modernization of enterprise data platforms using Databricks and cloud technologies. This individual will be responsible for building scalable, secure, and high-performing data environments from initial architecture and configuration through production deployment. They will design Databricks Lakehouse solutions, develop data pipelines, implement data governance, and help migrate legacy data systems into modern cloud platforms. The ideal candidate combines strategic architecture experience with strong hands-on engineering skills. They must be comfortable making technical decisions, writing code, configuring environments, troubleshooting complex issues, and leading engineering teams.

The biggest differentiator:

Our client needs someone who has personally led a complete Databricks implementation—not just someone who has worked within an established Databricks environment. Top 4 Required Skills 1. Databricks Architecture & End-to-End Implementation 3+ years of hands-on Databricks experience, including at least one enterprise implementation from initial architecture and environment setup through production. 2. Python, PySpark, Apache Spark & SQL Advanced data engineering skills, including building and optimizing distributed data pipelines and ETL/ELT solutions. 3. Cloud Data Architecture & Lakehouse Design Strong Azure, AWS, or GCP experience with Delta Lake, medallion architecture, cloud storage, and enterprise platform modernization. 4. Unity Catalog, Governance & Technical Leadership Experience implementing governance, security, access controls, data quality, architectural standards, and guiding engineering teams. Main Responsibilities Architect and deploy enterprise Databricks Lakehouse platforms from the ground up. Lead environment configuration, data integration, migration, testing, and production deployment. Design data pipelines using Python, PySpark, SQL, Apache Spark, and Delta Lake. Implement Bronze, Silver, and Gold data layers. Establish governance and security using Unity Catalog. Modernize legacy data warehouses and ETL platforms. Optimize compute performance, scalability, reliability, and cloud costs. Implement CI/CD pipelines and infrastructure automation. Develop architecture diagrams, standards, and technical roadmaps. Provide hands-on technical leadership to engineering, DevOps, analytics, and BI teams. Required Qualifications Qualification Requirement Overall experience 8+ years in data architecture, engineering, or warehousing Databricks 3+ years hands-on Implementation At least one complete Databricks implementation Programming Advanced Python, PySpark, and SQL Cloud Azure, AWS, or GCP Lakehouse Delta Lake, Unity Catalog, Spark Data engineering ETL/ELT, integration, modeling, optimization Modernization Legacy-to-cloud migration experience Leadership Architecture ownership, technical mentoring, stakeholder communication

Preferred:

Databricks certifications, Terraform, CI/CD, Azure Data Factory, AWS Glue, dbt, Airflow, Power BI, Tableau, streaming, and AI/ML workloads. What Success Looks Like The successful candidate delivers a production-ready Databricks platform, establishes secure and reusable architecture patterns, improves data processing performance, and enables engineering teams to build reliable analytics solutions independently.

Pay:

$100.00 - $130.00 per hour Expected hours: 40.0 per week

Benefits:

401(k) matching Health insurance Vision insurance

Experience:

Databricks:

3 years (Required)

Python:

3 years (Required)

Delta Lake:

2 years (Required)

Unity Catalog:

2 years (Required) Ability to

Relocate:

Kansas City, MO 64108: Relocate before starting work (Preferred)

Work Location:

Hybrid remote in Kansas City, MO 64108