Data Engineer

Data Engineer Must Have Technical/Functional Skills •

Orchestration:

Apache Airflow, Apache NiFi. •

Programming:

Java (Core), Python (for Airflow), Unix Shell Scripting. •

Big Data/Storage:

Apache Spark, Min

IO, AWS S3.

•

Security:

SSL/TLS, Certificate Management, IAM, Java Keystores. •

OS:

Linux/Unix (RHEL/Ubuntu).

Key Responsibilities:

•

Pipeline Orchestration:

Design and develop complex, reusable DAGs in Apache Airflow to automate data workflows, scheduling, and monitoring across the enterprise. •

Data Ingestion & Flow:

Create and optimize high-volume data streams using Apache NiFi, leveraging custom processors and controller services for diverse data sources and sinks. •

Custom Development:

Utilize Java to develop custom NiFi processors or troubleshoot the core NiFi framework, ensuring the platform meets specific architectural requirements. •

Object Storage Management:

Implement and manage data storage solutions using MinIO and AWS S3, ensuring high availability and efficient data retrieval patterns. •

Large-Scale Processing:

Develop and maintain Apache Spark jobs for heavy-duty data transformations, integrating them into NiFi and Airflow orchestration layers. •

Security & Certificate Management:

Secure NiFi clusters and data flows using TLS/mTLS, managing Java KeyStores (JKS), TrustStores, and SSL certificate lifecycles to ensure data-in-motion security. •

System Administration:

Perform environment setup, performance tuning, and troubleshooting within Unix/Linux environments, including shell scripting for task automation. •

Cloud Integration:

Deploy and manage data infrastructure components on AWS, utilizing IAM for access control and integrating cloud-native services with hybrid data pipelines. •

Monitoring & Optimization:

Establish robust logging and alerting for data pipelines to proactively identify bottlenecks, ensuring 99.9% reliability of data delivery. Salary Range- $100,000-$140,000 a year