A Data Engineer designs, builds and manages the information or big data infrastructure. Develops the architecture that helps analyze and process data in the way the organization needs it. Makes sure those systems are performing smoothly.
Employees in this job function are responsible for designing, building, and maintaining data solutions including data infrastructure, pipelines, etc. for collecting, storing, processing and analyzing large volumes of data efficiently and accurately
Key Responsibilities:
Collaborate with business and technology stakeholders to understand current and future data requirements Design, build and maintain reliable, efficient and scalable data infrastructure for data collection, storage, transformation, and analysis Plan, design, build and maintain scalable data solutions including data pipelines, data models, and applications for efficient and reliable data workflow Design, implement and maintain existing and future data platforms like data warehouses, data lakes, data lakehouse etc. for structured and unstructured data Design and develop analytical tools, algorithms, and programs to support data engineering activities like writing scripts and automating tasks Ensure optimum performance and identify improvement opportunities
Skills Required:
GCP Cloud Run, KAFKA, Cloud Architecture, Software Development, SQL, Cloud Computing, Big Data, Big Query, Application Development, Google Cloud Platform, Java, Application Testing, Agile Software Development, Artificial Intelligence & Expert Systems, Python, API Experience Required:
Engineer 3
Exp:
7+ years Data Engineering work experience
Education Required:
Bachelor's Degree Additional Information:
Spearhead the design, development, and maintenance of scalable data ingestion and curation pipelines from diverse sources. Ensure data is standardized, high-quality, and optimized for analytical use. Leverage tools and technologies, including Python, SQL, and DBT/Dataform, to build robust and efficient data pipelines. Utilize your full-stack skills to contribute to seamless end-to-end development, ensuring smooth and reliable data flow from source to insight. Leverage your deep expertise in GCP services (BigQuery, Dataflow, Pub/Sub, Cloud Functions, etc.) to build and manage data platforms that not only meet but exceed business needs and expectations. Implement and manage robust data governance policies, access controls, and security best practices to protect sensitive data. Employ efficient data workflow management and cloud infrastructure provisioning, championing best practices in Infrastructure as Code (IaC). Continuously monitor and improve the performance, scalability, and efficiency of data pipelines and storage solutions, ensuring optimal resource utilization and cost-effectiveness. Collaborate effectively with data architects, application architects, and cross-functional teams to define and promote best practices, design patterns, and frameworks for cloud data engineering. Proactively automate data platform processes to enhance reliability, improve data quality, minimize manual intervention, and drive operational efficiency. Clearly and transparently communicate complex technical decisions to both technical and non-technical stakeholders, fostering understanding and alignment. Stay ahead of the curve by continuously learning about industry trends and emerging technologies, proactively identifying opportunities to improve our data platform and enhance our capabilities. Develop comprehensive documentation for data engineering processes, promoting knowledge sharing, facilitating collaboration, and ensuring long-term system maintainability I'm interested
