Pay Growth
?
out of 5
Not enough data
Not enough info to score pay or growth
Find Jobs
Find Jobs Near You – Available Work in Your Location
Skip to job details
PL
Patton Labs Inc.
Senior Data Engineer
Review key factors to help you decide if the role fits your goals.
Job Security
?
out of 5
Not enough data
Calculating job security score...
Total Score
82
out of 100
Average of individual scores
Job Description
Design and implement scalable ETL frameworks to ingest, transform, and extract data from various source systems into Hive, PostgreSQL, and Redshift, generating business-driven extracts. Develop automated data ingestion pipelines for processing different file types (TXT, CSV) from AWS S3, dynamically loading them into Hive, PostgreSQL, Redshift, and Databricks based on project requirements. Lead database migration efforts, migrating SQL databases to AWS Redshift, leveraging Apache Spark for data transformation and AWS Athena for efficient querying of S3-stored data. Implement Type-1 and Type-2 Slowly Changing Dimensions (SCD) in ETL pipelines, ensuring historical data accuracy and compliance by handling overwrites (Type-1) and preserving versioned records (Type-2). Design and develop Spark-based ETL pipelines to migrate data from AWS Redshift to Databricks, applying complex transformations using PySpark/Scala and optimizing job performance. Work with AWS S3, Python, Scala, Spark, Spark