Data Engineering & Migration Lead

Data Engineering & Migration Lead Carmel, IN Job Details 1 day ago Qualifications Cloud identity and access management (IAM) Performance tuning Data storage Data transformation pipeline development Cloud data warehouses Automation Data migration Process optimization Implementing cloud migration projects HIPAA Snowflake SQL Technical solutions implementation Data Architecture Design (Architecture design skills) AWS Glue NIST standards Scripting System architecture design S3 Amazon CloudWatch Data validation Senior level Cloud monitoring

Leadership AWS Lambda Full Job Description Description:

At CSpring, we believe in the power of data to drive decisions and real-world impact. We're a purpose-driven consulting firm specializing in data strategy, data engineering, and data analytics. Our clients span the public and private sectors, and our work helps them solve complex problems, gain insights, and achieve measurable results. We're seeking talented professionals who are collaborative, curious, and committed to making a difference that thrive at the intersection of data, technology, and business strategy. Whether you're passionate about transforming public programs, enabling executive decision-making, or modernizing legacy systems, you'll find meaningful work and purpose here. Why You'll Love Working Here Purposeful Projects - Improve systems that serve real people by delivering smarter data, streamlined processes, and strategic insight. People-First Culture - We're as committed to your growth as we are to delivering high-impact solutions. You'll find support, autonomy, and community here. Strategic, Hands-On Work - From data architecture and documentation to client workshops and solution delivery, you'll influence every step of the process. Collaborative Trust - Our clients rely on us to listen carefully, deliver consistently, and guide wisely. We partner with integrity, curiosity, and heart. What You'll Do We are seeking a Data Engineering & Migration Lead to design, build, and execute the technical data migration strategy for a high-priority state agency application implementation. Unlike a standard ETL role, this position requires a "Migration Architect" mindset - someone who understands how to move massive, complex legacy datasets into a modern cloud environment using an incremental approach. You will be the primary technical owner of the migration pipeline, leveraging a Snowflake Medallion Architecture on AWS. You will lead the transition from legacy source systems through the "Bronze" (Raw) and "Silver" (Cleaned) layers, ensuring data integrity at every hop. 1.

Migration Pipeline Engineering Incremental Framework Design:

Develop and implement a robust, repeatable framework for incremental data loads to support a phased migration strategy.

AWS Ingestion:

Architect the "Bronze" layer ingestion process using AWS S3 and AWS Glue or Lambda, ensuring secure and efficient transfer of legacy data.

Snowflake Transformation:

Lead the development of SQL-based transformations within Snowflake to promote data from Bronze to Silver and Gold layers.

Performance Tuning:

Optimize ELT processes to handle large-scale data volumes while staying within agency cost and performance benchmarks. 2.

Technical Leadership & Strategy Technical Lead:

Serve as the subject matter expert for the migration engineering team, providing code reviews and architectural guidance.

Validation & Reconciliation:

Build automated reconciliation scripts to compare source and target counts/hash totals, ensuring zero data loss during the migration.

Error Handling:

Design "self-healing" pipelines with comprehensive logging, alerting, and retry logic for failed migration batches.

Collaboration:

Work closely with the PM/BA lead to translate functional mapping documents into executable technical logic.

Technical Environment Cloud Infrastructure:

AWS (S3, Lambda, IAM, CloudWatch)

Orchestration & ELT:

AWS Glue, AWS Lambda, or Step Functions Data Warehouse:

Snowflake (SnowSQL, Tasks, Streams, Snowpipe)

Architecture:

Medallion (Lakehouse) pattern with a focus on incremental loading

Requirements:

What You Bring You are a "builder" who understands that a migration is more than just a one-time script. You prioritize idempotency (the ability to run a process multiple times without changing the result), data lineage, and auditability, all of which are critical for state government compliance.

Migration Track Record:

Proven experience leading the technical implementation of at least two large-scale data migrations (Legacy-to-Cloud).

AWS Expertise:

Hands-on experience with AWS storage and compute services, specifically S3 and Glue or Lambda for data movement.

Snowflake Mastery:

Expert-level SQL skills and experience building multi-layer architectures within Snowflake.

Incremental Logic:

Deep understanding of Change Data Capture (CDC) or watermarking techniques for incremental migrations.

Public Sector Knowledge:

(Preferred) Experience working with state/federal data security requirements (e.g., NIST, CJIS, or HIPAA). Come Build With Us At CSpring, we unlock the potential of people and data. If you're ready to lead meaningful projects, collaborate with passionate teams, and grow your career in a people-first consulting environment - apply today!