Job Description
Job Description Everforth ECS is seeking a highly skilled Senior Data Engineer with extensive experience designing, building, and maintaining data platforms and pipelines for large-scale, enterprise applications in support of the Department of Homeland Security. Tools necessary for excelling in this role include SQL (PostgreSQL or MySQL preferred), Elasticsearch/OpenSearch, data modeling and transformation frameworks, and DevSecOps practices in the Cloud. This role will focus heavily on modernizing legacy data architectures and re-architecting the application's data layer, shifting from Elasticsearch to PostgreSQL or MariaDB as the system of record for transactional data. You will play a critical role in evolving scoring models, intelligence modifiers, and capability gap logic while ensuring alignment and consistency between the new relational database and Elasticsearch.
Key Responsibilities:
Design and architect robust, scalable data models, schemas, indexes, and stored procedures in a relational database to support transactional workloads and complex business logic. Lead the migration of the system of record from Elasticsearch to a relational database, including data modeling, transformation, validation, and cutover strategies. Develop and maintain data pipelines and integration processes to keep the database and Elasticsearch aligned for search and analytics use cases. Implement and evolve scoring models, intelligence modifiers, and capability gap logic in the data layer to support mission-critical decision support. Optimize database performance, including indexing strategies, query tuning, and partitioning to support high-volume, low-latency operations. Define and enforce data quality, data validation, and data governance standards across legacy and modernized components. Collaborate with application engineers to integrate APIs and services with the underlying data models and stored procedures, ensuring seamless functionality and consistent data semantics. Provide technical guidance and solutions for legacy data system modernization while minimizing downtime and operational risk during migration. Identify and remediate data-related technical debt in older systems while transitioning to modern data engineering patterns, tools, and frameworks. Integrate and operate CI/CD pipelines used for building, testing, and deploying data services and schema changes in cloud environments. Required Skills Ability to pass a Public Trust investigation. Must have ship. Bachelor's degree in a related field. Minimum 15 years general work experience. 8+ years of experience as a Data Engineer or in a closely related role, with strong experience in legacy data system modernization. 5+ years of hands-on experience with relational databases including from-scratch schema design, index creation, stored procedures SQL development, performance tuning, and transaction management. 3+ years of experience working with Elasticsearch or OpenSearch, including index design, mappings, and data ingestion strategies. Demonstrated experience designing and implementing data models and logic to support scoring models, rules engines, or similar decision-support systems. 5+ years of experience supporting applications in production, troubleshooting complex data issues in legacy environments, and proposing effective remediation strategies. Strong understanding of data modeling for OLTP/transactional systems, including normalization/denormalization tradeoffs and referential integrity. Solid understanding of Git, version control workflows, and collaborative development practices. Desired Skills Experience re-architecting systems to move the system of record between data stores (e.g., Elasticsearch to database), including coexistence, synchronization, and phased cutovers. Familiarity with automated testing frameworks and tools for data validation, regression testing, and schema change verification. Experience with the following tools:
Docker, Kubernetes, Jenkins, Terraform, Ansible, OpenSearch, Elasticsearch, Gitlab. 5+ years of experience with relational databases and ORM technologies (e.g., JPA/Hibernate), experience with Elasticsearch or OpenSearch alignment patterns a plus. Experience supporting analytics, reporting, monitoring, and/or data science workloads on top of transactional systems, including exposing data needed for scoring model evolution and intelligence modifiers. Familiarity working in Agile / SCRUM development environments.
