Senior Data Engineer / AI Engineer to design, develop, and optimize scalable data processing solutions and AI-driven applications. Strong expertise in PySpark, Python, Databricks, SQL, Agentic AI frameworks, and relational databases including Oracle and SQL Server Design, develop, and maintain large-scale data pipelines using PySpark and Databricks. Build efficient ETL/ELT solutions to ingest, transform, and process structured and semi-structured data. Optimize Spark jobs for performance, scalability, and cost efficiency. Develop reusable data engineering frameworks and best practices. Create and maintain data models, data marts, and analytical datasets. Implement data quality checks, monitoring, and governance controls. Design and develop Agentic AI solutions using LLM-based frameworks. Build autonomous AI agents capable of reasoning, planning, and task execution. Production streaming experience Pub/Sub and Dataflow, or Kafka / Flink / Kinesis including deduplication, ordering, and replay. Strong Python, advanced SQL, and Apache Beam. Deep hands-on
BigQuery:
partitioning, clustering, incremental merge patterns, cost-aware design. Dimensional modeling built in a real warehouse, including conformed dimensions. Dataform or dbt at production scale with tests or assertions and dependency management. Terraform, Git workflow, and CI/CD for data pipelines. Experience integrating a major SaaS platform as a data source. Track record leading a small team or owning a workstream, with judgment on which decisions are theirs and which belong to the architect. Production streaming experience Pub/Sub and Dataflow, or Kafka / Flink / Kinesis including deduplication, ordering, and replay. Strong Python, advanced SQL, and Apache Beam. Deep hands-on
BigQuery:
partitioning, clustering, incremental merge patterns, cost-aware design. Dimensional modeling built in a real warehouse, including conformed dimensions. Dataform or dbt at production scale with tests or assertions and dependency management. Terraform, Git workflow, and CI/CD for data pipelines. Experience integrating a major SaaS platform as a data source. Track record leading a small team or owning a workstream, with judgment on which decisions are theirs and which belong to the architect.
Required Qualifications:
Expertise in Scala (preferred) or Java, with proficiency in Python. Strong experience with Apache Spark for distributed data processing. Proficiency in building and maintaining ETL pipelines at scale.
Experience with AWS services:
S3, Glue, Athena, EMR Strong understanding of relational databases and SQL. Knowledge of data architecture, data warehousing, partitioning strategies, and columnar storage formats (e.g., Parquet). Experience implementing data quality checks and validation frameworks. Experience with workflow orchestration tools (Airflow preferred). Proficiency with Linux-based operating systems and shell scripting. Experience with Git-based version control and collaborative development workflows. Demonstrated ability and desire to continually expand skill set and learn from and teach others.
