Senior Data Engineer

Job Description:

We are looking for a highly skilled Senior Data Engineer with 8 to 12 years of experience in building and optimizing large-scale data processing pipelines. The ideal candidate will have strong hands-on expertise in Apache Spark using Java, proficiency in Databricks, and solid knowledge of Azure Data Lake Storage (ADLS) concepts and implementations. Experience with Python is also valuable, but Spark with Java is preferred.

Key Responsibilities:

Design, develop, and optimize scalable data pipelines and ETL processes using Apache Spark (Java preferred) on Databricks. Work closely with architects, data scientists, and business stakeholders to translate data requirements into scalable data solutions. Implement robust data storage and retrieval strategies using Azure Data Lake Storage (ADLS). Ensure high performance and reliability of Spark jobs, optimizing for memory, compute, and scalability. Integrate data from various sources and manage data ingestion pipelines effectively. Participate in code reviews, enforce best practices, and mentor junior team members. Collaborate with DevOps and cloud teams to ensure efficient deployment and monitoring of data pipelines.

Required Skills:

8 12 years of experience in Data Engineering or Software Development roles. Strong hands-on expertise in Apache Spark, preferably using Java (Python is a plus). Experience with Databricks for developing and managing Spark jobs. Deep understanding of Azure Data Lake Storage (ADLS) and its best practices. Experience in data modeling, schema design, and data performance tuning. Familiarity with big data tools and data lake architecture on Azure. Strong problem-solving, communication, and collaboration skills.

Preferred Skills:

Experience with CI/CD pipelines for data workflows. Exposure to Delta Lake, Azure Synapse, or other Azure-based data services. Certifications in Azure or Databricks would be an advantage.