Big Data Engineer

Big Data Engineer Allruva Technology Services Irving, TX Job Details Contract 22 hours ago Qualifications Spark SQL Spark implementation Data-driven problem-solving Full Job Description Job - ID 202402.TX0046 Excellent long- term contract opportunity for Big Data Engineer working on development of new cloud base applications in Dallas, TX.

Job Responsibilities:

Big Data Architecture and Design:

Design, implement, and maintain scalable and efficient big data solutions using Hadoop ecosystem components. Work closely with architects and data scientists to define data architecture and ensure alignment with business requirements.

Data Ingestion and Processing:

Implement data ingestion pipelines for large-scale data processing using tools such as Apache NiFi or custom scripts. Develop and optimize PySpark jobs for data processing and transformation.

Data Modeling and ETL:

Create and maintain data models to support efficient querying and reporting. Design and implement Extract, Transform, Load (ETL) processes for structured and unstructured data.

SQL Database Management:

Manage and optimize SQL databases for efficient storage and retrieval of structured data. Develop and maintain SQL scripts for data manipulation, querying, and reporting.

Performance Optimization:

Optimize PySpark and SQL queries for performance and scalability. Identify and resolve performance bottlenecks in big data processing.

Data Security:

Implement security measures for data at rest and in transit within the Hadoop ecosystem. Manage access controls and permissions for data stored in Hadoop and SQL databases.

Hadoop Cluster Administration:

Administer and maintain Hadoop clusters, ensuring high availability and reliability. Monitor and troubleshoot cluster performance and resource utilization.

Streaming Data Processing:

Implement and optimize real-time data processing using tools such as Apache Kafka and Spark Streaming. Develop and maintain streaming data pipelines for continuous data ingestion and analysis.

Data Governance and Quality:

Implement data governance policies and procedures to ensure data quality and integrity. Collaborate with data stewards to enforce data quality standards.

Documentation:

Create and maintain comprehensive documentation for data architecture, data flows, and system configurations. Document code, scripts, and processes for knowledge sharing and future reference.

Qualifications:

Education:

Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.

Technical Skills:

Proficiency in Hadoop ecosystem technologies, including HDFS, MapReduce, Hive, and HBase. Strong programming skills in PySpark for data processing and analysis. Advanced skills in SQL for data manipulation and querying.

Experience:

Several years of hands-on experience in big data engineering and analytics. Experience with end-to-end implementation of big data solutions.

Certifications:

Relevant certifications in Hadoop and Spark technologies (e.g., Cloudera Certified Data Engineer, Databricks Certified Developer).

Communication and Collaboration:

Strong communication skills to interact effectively with technical and non-technical stakeholders. Ability to work collaboratively in a team environment.

Problem-Solving:

Strong analytical and problem-solving skills. Ability to troubleshoot and optimize complex big data processes.

How To Apply:

Prospective candidates are invited to submit their resume and cover letter to