Pay Growth
?
out of 5
Not enough data
Not enough info to score pay or growth
Big Data Engineer
Job
Compunnel, Inc.
Costa Mesa, CA (In Person)
Full-Time
Review key factors to help you decide if the role fits your goals.
Job Security
?
out of 5
Not enough data
Calculating job security score...
Total Score
78
out of 100
Average of individual scores
Skill Insights
Compare your current skills to what this opportunity needs—we'll show you what you already have and what could strengthen your application.
Job Description
Job Summary The Big Data Engineer will be responsible for designing, developing, optimizing, and supporting scalable big data platforms and cloud-native applications within AWS environments. This role requires expertise in both batch processing and API-driven architectures, leveraging technologies such as Spark, Hadoop, Scala, PySpark, Java, and Snowflake. The ideal candidate will have strong experience building enterprise-grade data pipelines, microservices, and cloud-based analytics solutions while ensuring performance, scalability, and reliability. Key Responsibilities
- Design, develop, implement, test, and maintain Big Data applications supporting both batch and API-based workloads.
- Build and optimize scalable data ingestion, transformation, and processing pipelines using Spark and related technologies.
- Develop and maintain cloud-native applications and data platforms within AWS environments.
- Design, develop, and support REST APIs and SOAP web services for data and application integration.
- Build and maintain microservices architectures using AWS services and modern development frameworks.
- Develop scalable batch processing solutions using Hadoop, Spark, EMR, and distributed computing technologies.
- Optimize data processing jobs and applications for performance, scalability, and cost efficiency.
- Develop and support data workflows using AWS Step Functions, Apache Airflow, and related orchestration tools.
- Design and implement data ingestion and transformation pipelines within Snowflake environments.
- Work with large-scale structured and unstructured datasets across distributed systems.
- Develop automation solutions using Python, shell scripting, and cloud-native tools.
- Support and maintain Cassandra databases and distributed data storage solutions.
- Troubleshoot production issues and perform root cause analysis across data platforms and applications.
- Collaborate with cross-functional teams to gather requirements and deliver scalable solutions.
- Participate in code reviews, testing, deployment activities, and continuous improvement initiatives.
- Maintain technical documentation and operational procedures. Required Qualifications
- 7+ years of experience in the analysis, development, implementation, and testing of Big Data applications.
- Strong experience working within AWS cloud environments.
- Strong programming experience with Scala.
- Strong experience with PySpark and Apache Spark.
- Strong Java development experience, including REST APIs and SOAP web services.
- Experience with Hadoop ecosystem technologies including Hadoop, HDFS, Spark, and MapReduce.
- Hands-on experience with Cassandra.
- Experience building and supporting applications using AWS services including:
- Amazon EMR
- Amazon EC2
- Amazon ECS
- Amazon S3
- AWS Step Functions
- API Gateway
- Experience working with both batch processing systems and API-driven microservices architectures.
- Strong experience with performance tuning and optimization of Spark, Hadoop, and EMR workloads.
- Strong Linux administration and troubleshooting experience.
- Experience with shell scripting and Python automation.
- Experience building data ingestion and transformation pipelines using Snowflake and Spark.
- Strong analytical, troubleshooting, and problem-solving skills.
- Ability to work independently in a remote environment.
- Strong communication and collaboration skills. Preferred Qualifications
- Experience with Python and Kotlin development.
- Experience with Apache Airflow for workflow orchestration.
- Experience with DevOps tools including Git, GitHub, GitLab, and Bitbucket.
- Experience with development tools such as IntelliJ IDEA and PyCharm.
- Experience implementing CI/CD pipelines using Maven, Gradle, Jenkins, and Artifactory.
- Experience with AWS CodeCommit and AWS CloudFormation.
- Experience with cloud-native architecture patterns and distributed systems design.
- Experience supporting enterprise-scale analytics and data platforms.
- Knowledge of data governance, security, and operational best practices. Required Skills
- AWS (EMR, EC2, ECS, S3, Step Functions, API Gateway)
- Scala
- PySpark
- Java
- Hadoop
- HDFS
- Apache Spark
- MapReduce
- Cassandra
- Snowflake
- Linux
- Shell Scripting
- Python
- REST APIs
- SOAP Web Services
- Data Pipeline Development
- Batch Processing
- Performance Tuning
- Microservices Architecture