Senior Databricks Architect / Lead Data Engineer Remote in Westchester, IL, US

• Posted 46 minutes ago

• Updated 22 minutes ago Full Time Part Time On-site USD $50-60/hr Fitment Dice Job Match Score™ 🔢 Crunching numbers... Job Details Skills Data Engineering Data Engineering Summary Position Summary We are seeking an experienced Senior Databricks Architect / Lead Data Engineer to lead the design, development, and implementation of modern data platforms leveraging Azure Databricks . The ideal candidate will possess deep expertise in building scalable data engineering solutions, strong hands-on coding skills, and proven experience delivering enterprise-grade Databricks implementations for clients. This role requires a blend of technical leadership, architecture, solution delivery, and client-facing consulting capabilities. Key Responsibilities Solution Architecture & Design Design and architect scalable, secure, and high-performance data platforms using Azure Databricks . Lead the design and implementation of modern data lakehouse architectures utilizing Delta Lake , Unity Catalog , and Databricks best practices. Define data ingestion, transformation, orchestration, governance, and consumption patterns across enterprise data ecosystems. Establish architecture standards, development frameworks, and reusable components. Data Engineering & Development Develop and optimize complex data pipelines using PySpark and Databricks. Build and maintain ETL/ELT frameworks leveraging Delta Live Tables (DLT) and Databricks Workflows . Ensure code quality, performance optimization, scalability, and reliability of data processing solutions. Implement data quality, monitoring, lineage, and governance controls. Project Delivery Lead end-to-end Databricks implementations from strategy and design through deployment and stabilization. Collaborate with business stakeholders, data consumers, architects, and platform teams to translate business requirements into technical solutions. Guide development teams and perform design/code reviews. Troubleshoot complex technical challenges and provide solution recommendations. Client Engagement & Consulting Serve as a trusted advisor to client stakeholders on data modernization and analytics transformation initiatives. Conduct architecture workshops, solution demonstrations, and technical assessments. Present technical solutions and recommendations to business and executive audiences. Support estimation, solutioning, proposal development, and pre-sales activities as needed. Required Qualifications 10+ years of overall experience in Data Engineering, Data Architecture, or Analytics platform delivery. 5+ years of hands-on experience with Databricks . Demonstrated experience delivering at least two (2) end-to-end Databricks implementations in enterprise environments. Strong hands-on coding expertise in PySpark with the ability to develop solutions during technical interviews and coding assessments.

Extensive experience with:

Unity Catalog Delta Lake Delta Live Tables (DLT) Databricks Workflows Strong experience designing and implementing Azure Databricks architectures . Experience integrating Databricks with Azure services such as Azure Data Lake Storage (ADLS), Azure Data Factory (ADF), Azure Synapse, Azure Key Vault, and Microsoft Fabric (preferred). Strong understanding of data modeling, ETL/ELT design patterns, data governance, and performance tuning. Proven client-facing consulting experience with excellent communication and stakeholder management skills. Preferred Qualifications Databricks Certified Professional or Associate certifications. Azure Data Engineering and/or Azure Solutions Architect certifications. Experience implementing CI/CD, Infrastructure as Code, and DevOps practices for Databricks environments. Experience with data governance, metadata management, security, and access controls in enterprise data platforms. Familiarity with streaming architectures, real-time analytics, and advanced analytics workloads. Key Competencies Enterprise Data Architecture Data Lakehouse Design PySpark Development Azure Databricks Solution Leadership Client Relationship Management Technical Consulting Data Governance & Security Problem Solving & Analytical Thinking Team Leadership & Mentoring Employers have access to artificial intelligence language tools ("AI") that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.

Dice Id:

10485596

Position Id:

TYN - 13491-13495-1788374338

Posted 46 minutes ago