Lead Data Engineer

Day to

Day Job Duties:

Perform end-to-end datastore migration from on-prem Data Lake to AWS-hosted Lakehouse as part of the migration factory team Refactor and migrate data pipelines including extraction logic and job scheduling. Execute data transfer ensuring integrity and reliability. Translate legacy SQL and Spark logic to Snowflake and Iceberg. Analyze data usage patterns to deliver data products. Build data reconciliation frameworks. Collaborate with stakeholders for validation and sign-off. Act as technical liaison. Troubleshoot and optimize pipelines. Adapt to new tools and workflows Follow SDLC and CI/CD practices. Collaborate with global teams.

Basic Qualifications:

5 7 years hands-on coding Strong SQL troubleshooting Python or Java proficiency Apache Spark experience ETL/ELT pipeline experience CI/CD and SDLC familiarity Kubernetes exposure Relevant degree

Core Data Engineering Competencies:

SCD Type 2 Schema evolution (Iceberg) Partitioning and clustering Normalization vs denormalization Natural vs surrogate keys Data quality frameworks

Technical Stack Requirements:

Kafka, ANSI SQL, FTP, Apache Spark JSON, Avro, Parquet Hadoop, Snowflake, Iceberg, Sybase IQ Core Competencies:

Integrity Collaboration Communication Stakeholder management Global teamwork Ownership Professionalism Continuous improvement Nice to

Have:

Migration experience Financial services domain Lakehouse experience