Data Engineer II

We are looking for a Data Engineer to join the Infrastructure Automation team. Our success depends on our world-class network and hardware infrastructure; we’re handling massive scale and rapid integration of emergent technologies. Our goal is to become “The Infrastructure Platform” for the world. The Infrastructure Automation team is responsible for delivering the software that powers our infrastructure. As a Senior DE you will create solutions to integrate with multi heterogeneous data sources, aggregate and retrieve data in a fast and safe mode, curate data that can be used in reporting, analysis, machine learning models and ad-hoc data requests. You should have excellent business and communication skills to be able to work with business owners, Product teams and Tech leaders to gather infrastructure requirements, design data infrastructure, build up data pipelines and data-sets to meet business needs. You will be responsible for designing, developing, and operating a data service platform using Python, Airflow, and SQL to build the various ETL, analytics, and data quality components. You’ll automate deployments using AWS CodeDeploy, AWS CodePipeline, AWS Cloud Development Kit (CDK), and AWS Cloud Formation. You will work with AWS services like Redshift, Glue, S3, IAM, CloudWatch, and more. Responsibilities

• Interface with other technology teams to extract, transform, and load data from a wide variety of data sources using Python, SQL and AWS big data technologies.
• Explore and learn the latest AWS technologies to provide new capabilities and increase efficiencies of the team.
• Designing and implementing complex pipelines and other Data Engineering solutions.
• Work closely with business owners, developers, Business Intelligence Engineer to explore new data sources and deliver the data.
• Create extensible designs and easy to maintain solutions with the long term vision in mind
• Improve tools, processes, scale existing solutions, create new solutions as required based on team and stakeholder needs
• You will work with business customers in understanding the business requirements and implementing solutions to support analytical and reporting needs.
• A good candidate can partner with business owners directly to understand their requirements and provide data which can help them observe patterns and spot anomalies. Required Skills & Experience
• 5+ years of related experience.
• Expertise with data modeling, warehousing and building ETL pipelines
• Experience in at least one modern scripting or programming language, such as Python, Java, Scala, or NodeJS
• Should have experience developing complex and a variety of reports.
• A good candidate has strong analytical skills and enjoys working with large complex data sets.
• Expert knowledge of SQL Preferred
• Experience with big data technologies such as: Hadoop, Hive, Spark, EMR
• Experience operating large data warehouses

Compensation:

$75/hour We look forward to reviewing your application. We encourage everyone to apply

• even if every box isn’t checked for what you are looking for or what is required.

PDSINC, LLC

is an Equal Opportunity Employer.