W2 profile only
Location:
Whippany, NJ (Hybrid)
Interview Mode:
In person interview required
Job Description:
- H ands‑on senior engineer , working directly with developers on design and implementation of modernization initiatives.
- Strong Data engineer with more than 8 years of experience
- Strong hand on experience in Python
- Strong handoff experience in Pyspark and stream processing with Kafka.
- Lead containerization and cloud onboarding of services.
- Drive adoption of GitLab
CI/CD, M1
pipelines, and release automation .
- Champion modern testing practices
- Drive Kafka adoption for event driven design standards
Role Overview Pipeline Development :
Build and maintain ETL/ELT pipelines for ingesting and transforming data.
Data Warehousing :
Design and manage data warehouses and lakes (Snowflake, BigQuery, Redshift).
Big Data Processing :
Optimize large-scale data workflows using Apache Spark or Hadoop.
Data Governance :
Ensure data quality, lineage, and compliance with regulations.
Workflow Orchestration :
Use Airflow or similar tools to schedule and monitor pipelines.
Integration :
Connect APIs, databases, and streaming sources (Kafka).
Collaboration :
Partner with analysts, data scientists, and business teams to deliver usable datasets. Required Skills Skill Area Key Tools/Technologies Why It Matters Programming Python, SQL, Scala, Java Core for building pipelines and transformations Databases MySQL, PostgreSQL, MongoDB, Cassandra Supports structured and unstructured data Big Data Apache Spark, Hadoop, Kafka Enables processing of massive datasets ETL Tools Airflow, dbt, Talend, Informatica Automates and manages workflows Cloud Platforms AWS (Glue, Redshift, S3), Azure (Synapse, Data Lake), Google Cloud Platform (BigQuery) Provides scalability and cost efficiency Data Modeling Star/Snowflake schemas, partitioning Ensures optimized storage and query performance Security Role-based access, encryption Critical for compliance and governance
Risks & Challenges Data Quality Issues:
Poor validation can lead to unreliable analytics.
Pipeline Failures:
Inadequate monitoring may cause downtime and data loss.
Cost Overruns:
Inefficient queries or storage can inflate cloud costs.
Compliance Risks:
Missing
GDPR/DPDP
controls can lead to legal exposure. Best Practices Automate pipeline monitoring with Airflow/Kafka . Use data profiling before ingestion to detect anomalies. Implement partitioning and indexing for performance. Collaborate closely with data science teams to align schema design.
