Senior Data Fabric Engineer

Role Overview We are seeking a highly skilled Senior Data Engineer with a deep specialization in Apache Spark and the Microsoft Fabric ecosystem. The ideal candidate isn''t just a notebook user; you are an expert in developing, packaging, and deploying Spark-based JAR files (Scala/Java) to handle complex, high-scale data processing requirements. You will be responsible for architecting robust ETL/ELT pipelines, optimizing Spark performance, and leveraging the full suite of Microsoft Fabric tools (OneLake, Lakehouse, and Data Factory) to drive our data strategy forward.

Key Responsibilities Custom Spark Development:

Design and develop high-performance data processing applications using Scala or Java, compiled into JARs for execution on Spark clusters.

Fabric Implementation:

Architect and maintain end-to-end data solutions within Microsoft Fabric, utilizing Synapse Data Engineering and OneLake.

Performance Tuning:

Optimize Spark jobs by managing partitions, caching strategies, and memory management to ensure efficient resource utilization.

Pipeline Orchestration:

Build and automate sophisticated data workflows using Fabric Data Factory and Airflow (or similar).

DevOps & CI/CD:

Manage the lifecycle of Spark JARs through automated CI/CD pipelines (Azure DevOps/GitHub Actions), ensuring seamless deployment to Fabric environments.

Data Modeling:

Implement Medallion Architecture (Bronze/Silver/Gold) and maintain Delta Lake tables for ACID compliance and time travel capabilities.

Technical Requirements Core Essentials Spark Expertise:

5+ years of experience with Apache Spark, specifically in writing and deploying compiled JAR files rather than solely relying on PySpark notebooks.

Languages:

Proficiency in Scala or Java (required) and Python/SQL (preferred).

Platform:

Hands-on experience with Microsoft Fabric or migrations from Azure Databricks/Synapse to Fabric.

Storage:

Deep understanding of Delta Lake format and Parquet optimization.