Job Description Help for Job Description. Opens a new window. Senior Data Engineer-Remote Multiple positions available

Requirements:

Bachelors degree in Computer Science, Mathematics, Statistics or IT related field 5+ years professional data engineering related experience Obtain and maintain a Secret security clearance issued by the Department of Defense (DoD). United States Citizenship is required

Skills:

Professional experience in data architecture, data engineering, data hub, data lake, and/or data warehouse development. Experience with Databricks and/or Palantir Foundry required Active CompTIA Security+ or CompTIA Network+ certification preferred. If selected, the candidate must be able to obtain a CompTIA Security+ certification prior to being eligible to begin supporting the program.

Work Schedules:

IBR promotes work-life balance by offering flexible scheduling options. Standard business hours are aligned to the Eastern Time Zone. Responsibilities Provide data engineering expertise in the development, implementation, integration, and sustainment of data architectures, data hubs, data lakes, and data warehouse solutions. Support the design and development of data models, data structures, and data acquisition processes that enable data-driven decision making across the HC/HR Data Domain. Develop engineering and implementation plans for data hubs, data acquisition, data modeling, data integration and related data engineering activities. Research and evaluate existing data sources within the data lake and enterprise data environment to identify authoritative, reliable, and appropriate sources for data hub development. Plan, create, and maintain data architectures, ensuring alignment with business requirements. Design, develop, maintain, and optimize ETL/ELT data pipelines and data transformation processes supporting enterprise data integration and data hub activities. Develop and maintain batch and streaming data pipelines using technologies such as Spark, Python, Databricks, Palantir Foundry, Kafka, and related data engineering tools. Develop and maintain data acquisition processes to ingest, transform, validate, and integrate data from diverse enterprise sources. Implement incremental data loading strategies to optimize data processing and data freshness. Define and implement approaches for handling late-arriving data, processing windows, data freshness, and other data lifecycle considerations. Identify opportunities to automate manual data processes and improve the efficiency, reliability, and scalability of data engineering workflows. Develop, maintain, and optimize data engineering solutions within Databricks and enterprise data lake environments. Configure, monitor, and manage Databricks clusters in accordance with DON policies, standards, security requirements, and technical guidelines. Document Databricks cluster configurations, parameters, and operational standards for internal and external stakeholders. Develop and maintain Spark-based data processing solutions using PySpark, Spark SQL, Spark Data Frame, Data Sets, and related technologies. Implement and maintain Delta Lake solutions, including Delta Live Tables where applicable, to support scalable and reliable data processing. Optimize data processing, storage, and query performance within data lake environments. Support data engineering and application development activities within Palantir Foundry, including development and maintenance of ontologies, ETL/ELT pipelines, applications, and data-driven user interfaces. Support the integration of Palantir Foundry capabilities with enterprise data engineering and analytics environments. Develop and maintain streaming data solutions using Kafka, Kafka Streams, ksqlDB, and related technologies. Configure and manage Kafka topics and associated components, including Schema Registry, to support reliable and scalable data processing. Develop and maintain Python-based data processing applications and AWS Lambda functions. Support data integration and processing using AWS services such as S3, Kinesis, Lambda, and DynamoDB. Monitor, troubleshoot, and optimize cloud-based data processing solutions for performance and scalability. Develop and maintain data quality controls, validation processes, and data quality gates to ensure data accuracy, completeness, consistency, and reliability. Develop and data-driven testing and unit testing for Spark, Python, and other data processing solutions. Establish and maintain data lifecycle policies and processes, including retention, backup, recovery, and data management requirements. Monitor and troubleshoot data pipelines and processing jobs to identify and resolve data quality, performance, and integration issues. Identify opportunities to improve data processing performance, reliability, scalability, and maintainability.

Application Deadline:

Applications will be reviewed as received and accepted on a continuous basis until the position is filled. Early application is strongly encouraged. Benefits and Compensation Nationwide medical, dental, and vision insurance 3 weeks of Paid Time Off and 11 Paid Federal Holidays 401k matching Life Insurance, Short-Term Disability, and Long-Term Disability at no cost to our employees Supplemental insurance options Flexible spending accounts and Dependent Care spending accounts Wellness incentives Reimbursement for professional development and certifications Access to training assistance opportunities to support career growth and progression Hybrid and Remote work opportunities to support work-life balance

Salary Range:

$115,000.00 - $165,000.00 USD annually. Equal Employment Opportunity (EEO) Statement Imagine Believe Realize, LLC (IBR) is proud to be an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to legally protected characteristics such as race, ethnicity, color, religion, sex, sexual orientation, gender identity, gender expression, creed, age, national origin, disability (including pregnancy, childbirth, or related medical conditions), marital status, veteran status, or any other applicable legally protected class in accordance with federal, state, and local laws.