Lead Data Engineer

Only local San Francisco Bay Area candidates! Direct W2 contractors only! No 3rd party agencies!

Hybrid contract:

3 days onsite and 2 days remote in South San Francisco. No Relocation possible. We don't sponsor any kind of Visa! As a Lead Data Engineer, you will act as a hands-on technical leader- designing and building data solutions while guiding engineering best practices across the team. This is a player-coach role, where you will actively contribute to development while mentoring others and driving high-quality delivery. You will be a pivotal member of our team, responsible for:

Key Responsibilities:

Hands-on Data Engineering & Delivery:

Design, build, and maintain scalable data pipelines to ingest, transform, and curate structured and unstructured data. Write production-quality code in SQL and Python and actively contribute to day-to-day development. Troubleshoot, optimize, and improve performance of data workflows and systems.

Technical Leadership & Data Architecture Ownership:

Lead by example through hands-on contributions to critical projects. Provide technical guidance, code reviews, and mentorship to other data engineers. Help drive implementation of best practices in data engineering, testing, and deployment.

Design and Build Scalable Data Pipelines:

Architect, develop and oversee development of pipelines to ingest, transform, and curate structured and unstructured data from internal and external sources. Ensure high performance, scalability, and reliability of data systems

Data Profiling, Mapping & Standardization:

Profile data, identify quality issues, and align disparate datasets. Define data models and standardization frameworks to support scalable, reusable, and AI/ML-ready data products.

Data Product Engineering & API Development:

Build and maintain reusable data products and APIs to support analytics and AI use cases. Ensure solutions are well-documented, secure, and scalable.

AI/ML Enablement:

Work closely with data scientists to prepare and deliver high-quality datasets for ML and AI use cases. Support data pipelines for LLM and AI-driven workflows.

Metadata Management & Data Governance:

Champion data governance, lineage, and metadata management practices Ensure compliance with enterprise data security and privacy standards.

Monitoring and Event Frameworks:

Implement monitoring, alerting, and event-driven frameworks for data pipelines. Ensure robustness, observability, and reliability of data systems.

Container and Workflow Orchestration:

Lead adoption of containerized and orchestrated data workloads (e.g., Docker, Amazon EKS) Guide orchestration of complex AI/data workflows using modern tooling. Cross-functional Collaboration & Influence Partner with business, product, and external stakeholders to align data strategy with organizational goals. Translate business needs into scalable technical solutions.

Continuous Improvement:

Identify opportunities to improve tooling, processes, and performance. Stay hands-on with new technologies and bring practical improvements to the team.

Qualifications Basic Qualifications:

Bachelor's or Master s degree in Computer Science, Engineering, Data Science, or a related technical field. 7+ years of experience in data engineering or similar roles, including experience leading projects or initiatives. Proven track record designing and scaling cloud-based data platforms (preferably AWS). Strong proficiency in SQL, Python, and advanced data modeling techniques. Experience leading architecture decisions and implementing best practices. Strong understanding of data quality, integration, transformation, and governance. Excellent communication skills with the ability to influence technical and non-technical stakeholders

Preferred Qualifications:

Experience acting as a technical lead or senior individual contributor on data engineering projects. Hands-on experience with AWS data services (Glue, Redshift, S3, Lambda, Athena, etc.). Experience supporting AI/ML data pipelines and workflows. Familiarity with metadata management and data governance frameworks. Experience in healthcare/life sciences or partnering domains. Experience working in Agile environments.