Job Description
Sr. Data Engineer#26-17199
$76.44-$80.54 per hour
ST Paul, MN
Onsite
Job Description
Our client, an Agriculture and Energy Services company, is looking for a Sr. Data Engineer for their St Paul, Minneapolis or Grove Heights location.
Responsibilities:
Support a variety of projects that will range in scale and complexity within the Enterprise Big Data Ingestion Team. Play a critical role in designing, developing, and supporting our data ingestion pipelines, ensuring seamless integration of data from client various sources to support robust analytics and business intelligence.
Cloud Data Platforms:
Utilize cloud data warehouses using Snowflake and AWS, ensuring optimal performance, security, and scalability of data pipelines.
Python Development:
Develop, optimize, and maintain data processing scripts and applications using Python to support various big data ingestion tasks.
Adhere to, improve, and contribute perspective to programming and development deadlines for the team.
ELT Process Management:
Assist with architecting and implementing of Extract-Load-Transform (ELT) processes, including AWS Lambda functions, to ensure efficient and accurate data loading between systems.
Data Transformation:
Utilize Snowflake's functionality of tasks, streams, stored procedures, dynamic tables, and dbt (data build tool) to model raw data into a consumable format for data engineers, analysts, and scientists.
Business Integration:
Understand business processes, data flows, and strategies for Client business units and work collaboratively on their data needs.
Documentation and Standards:
Produce, maintain, and encourage high-quality documentation to ensure transparency and understanding of data ingestion pipelines and solutions.
Monitoring and Observability:
Understand, monitor, and enhance data ingestion flows to enable reporting and insights development, including tracking data and job statuses.
Team Leadership:
Mentor and guide other team members, fostering a culture of continuous learning and development.
Advise on best practices around data ingestion pipelines and procurement of data based on use-cases (structured, semi-structured, and unstructured data).
Requirements:
Top 3 Technical Skills:
Python
SQL
Writing Lambdas
Top 3 Functional Skills:
Understanding how loans work
Gathering requirements
Can work with little direction
Bachelor or Graduate degree in Business Management, Information Management, Computer Science, or other STEM degree program.
5+ years in relevant data engineering positions.
Strong understanding of modern, cloud-based data architecture concepts and best practices, including Lakehouse architectures.
Hands-on experience using version control systems such as Git and CI/CD workflows and practices.
Comfortable with ambiguity; able to take ownership, thrive with less oversight and process.
Experience in ELT/ETL tools, including AWS Glue, AWS Lambda, and Apache Airflow.
Experience leading other team members and developing technical standards.
Attention to detail and commitment to data and code quality.
Strong analytical and problem-solving skills and an ability to multi-task.
Experience with data observability and monitoring tools, such as Dynatrace, Datadog, Splunk, Monte Carlo, and AWS CloudWatch.
