A Data Engineer designs, builds and manages the information or big data infrastructure. Develops the architecture that helps analyze and process data in the way the organization needs it. Makes sure those systems are performing smoothly.
The Senior Data Engineer at Satsyil Corp will be responsible for designing, implementing, and optimizing enterprise data architectures using Databricks as the primary data lakehouse platform. This role requires expertise in Databricks platform capabilities, medallion architecture patterns, and modern data engineering practices to deliver scalable, governed, and performant data solutions supporting analytics, AI/ML, and business intelligence initiatives.
Responsibilities:
Design and implement end-to-end data lakehouse architectures using Databricks on AWS/Azure, incorporating medallion architecture patterns for progressive data refinement Architect Delta Lake storage solutions with appropriate partitioning strategies, Z-ordering, and optimization techniques for high-performance query workloads Design Unity Catalog implementations for centralized data governance, including metastore configuration, catalog/schema hierarchies, cross-workspace data sharing, and fine-grained access controls Design parameterized, idempotent, and auditable data pipelines using Databricks workflows, Delta Live Tables, Apache Spark, Structured Streaming, and Auto Loader Implement data quality frameworks with automated validation checks, anomaly detection, data profiling, and data lineage tracking integrated into pipeline workflows Establish encryption strategies, audit logging frameworks, and compliance controls aligned with regulatory requirements Optimize Databricks cluster configurations, query performance, and implement cost management strategies using photon acceleration, serverless compute, and resource tagging Architect feature stores, MLOps pipelines with MLflow, and analytical data models optimized for BI tools and self-service analytics
Experience working with FISMA, FedRAMP, HIPAA, or other regulatory requirements Knowledge of data encryption strategies and audit logging frameworks Experience optimizing Databricks cluster configurations and query performance
