Senior Data Engineer Position Available In Fulton, Georgia

The Data Engineer position works independently on client engagements, takes part in the development of our practice, aids in business development, and contributes innovative ideas and initiatives to our company. They serve as a trusted advisor working together with our clients, from data owners and analytic users to C-level executives. They work independently as part of a small team to solve complex data engineering use-cases across a variety of industries.

Day-to-day responsibilities include the following

• Design and develop analytical layers, building cloud data warehouses, data lakes, ETL/ELT pipelines, and orchestration jobs
• Work with modern tools such as Snowflake, Databricks, Fivetran, and dbt and credentialize your skills with certifications
• Write code in SQL, Python, and Spark, and use software engineering best-practices such as Git and CI/CD
• Support the deployment of data science and ML projects into production.

(

Note:

You will not be developing machine learning models or algorithms.

• Provide technical expertise in developing the data architecture in Azure and migration of data ingestion and retrieval to Databricks
• Design and implement highly scalable solutions in Databricks to ensure data retrieval from Alteryx and PowerBI
• Lead the development of stored procedures in SQL for the organization to access different mill data in Azure
• Design and implement best practices in maintaining code repositories (Git), parameterization, coding conventions and nomenclature in both Azure and Databricks
• Optimize and automate data pipelines in Databricks using PySpark to ingest, transform data and store in delta tables and ADLS container to increase efficiency and minimize operating costs
• Create the data model for plate mill data from raw to production ready data for analysts to consume
• Write custom SQL procs for translation of non-English tables to English tables
• Collaborate with the DevOps and security teams to create scalable, secure, cost-effective and automated cloud-based infrastructure to host different products and solutions
• Create datamarts and tables that can easily be accessed by the analytic engineering and data analysts in PowerBI and Alteryx
• Document and review findings, architecture designs and infrastructure designs
• Analyze and transform data collected from different mills about the different treatments that it undergoes, mill run time, cost and profit analysis at mill and company level to ensure that its clean and formatted and ready for consumption by PowerBI
• Create a roadmap for the different phases of migration/development from Azure to Databricks
• Create analytical content and blogs for the company website showcasing various data modeling methods, CI/CD pipelining tool comparisons and best practices
• Develop and create products in Snowflake that can be published on the marketplace for product development.