Compare your current skills to what this opportunity needs—we'll show you what you already have and what could strengthen your application.
Job Description
Job Title:
Azure Databricks Architect Location:
Remote with (50% travel) Role Summary We are seeking Senior Data Platform Engineers to join our CSO team. This is a dual-domain environment spanning two fully segregated platforms built on Azure Databricks and the
Medallion Architecture:
the CSO Security Data Platform and the SOX Application Data Platform (fully segregated to satisfy Sarbanes-Oxley requirements). Engineers are the backbone of both platforms - ensuring data quality, enabling security consumers, onboarding new sources, maintaining pipelines, and supporting SOX audits.
Required Qualification:
Experience 5+ years of data engineering experience with at least 2 years on Azure Databricks or equivalent cloud lakehouse platforms. Hands-on experience with Medallion Architecture pipelines, Delta Lake (MERGE, OPTIMIZE, VACUUM, Z-ORDER, schema enforcement), and Databricks Structured Streaming. Familiarity with SOX compliance, IT General Controls (ITGC), or regulated-environment data engineering; experience supporting audits or evidence collection. Technical Skills Area Technologies Languages Python (PySpark), SQL; Scala a plus Cloud Azure (ADLS Gen2, Azure Monitor, Log Analytics, Event Hubs, APIM, Azure DevOps) Data Platform Databricks (Unity Catalog, Auto Loader, Structured Streaming, Workflows, Genie, AI/BI Dashboards, MLflow) Governance Unity Catalog (RBAC, column/row security, lineage), MS Purview (auto-classification, sensitivity labels) Integration Azure APIM, Delta Sharing, Databricks
SQL API, SIEM/SOAR
connector patterns BI & Reporting Databricks AI/BI Dashboards, Power BI DevOps Git, CI/CD via Azure DevOps or GitHub Actions Security Domain Knowledge Familiarity with OCSF (Open Cybersecurity Schema Framework) or willingness to develop deep expertise rapidly. Understanding of security log source types (EDR, firewall, IAM/PAM, CASB) and
SOC/SIEM/SOAR
workflows; awareness of
MITRE ATT&CK
framework. Preferred Qualifications OCSF normalization implementation across multiple security log sources. Experience with Azure API Management (APIM) for data platform API governance. UEBA, threat scoring, or ML-backed security analytics (MLflow, Databricks Feature Store, Mosaic AI). SOX audit participation - auditor walkthroughs, evidence packages, IT General Controls testing.
Certifications:
Databricks Certified Data Engineer Associate/Professional, Azure Data Engineer Associate (DP-203), or equivalent. Graph-based correlation (Databricks GraphX); Delta Sharing. Core Competencies Competency Why It Matters Ownership & Reliability Numerous sources, high volume TB/day - you own what doesn''t run. Audit Mindset Audits mean evidence gaps have real regulatory consequences. Security Domain Curiosity The better you understand analyst needs, the better the platform serves them. Stakeholder Communication You bridge data engineering, SOC, compliance, and auditors. Structured Problem-Solving Quarantine spikes and pipeline failures require root-cause discipline. Documentation Discipline Runbooks, audit evidence, and onboarding guides are first-class deliverables. Compliance Awareness SOX platform changes carry regulatory risk; process adherence is non-negotiable. Platform & Tooling Landscape Area Technologies Cloud Infrastructure Microsoft Azure, ADLS Gen2, Azure Monitor, Log Analytics, Event Hubs, APIM Data Platform Databricks (Unity Catalog, Structured Streaming, Auto Loader, Workflows, Genie, AI/BI Dashboards) Storage / Format Delta Lake, JSON + GZip (Landing Zone), ADLS Gen2 Hot & Archive tiers Schema Standard OCSF + custom.• namespace (Security); segregated schemas (SOX) Governance Unity Catalog, MS Purview (auto-classification, sensitivity labels, federated discovery) Integration Azure APIM, Delta Sharing, Event Hubs, Databricks
SQL API BI
& Analytics Databricks AI/BI Dashboards, Databricks Genie, Power
BI AI & ML
MLflow, Databricks Feature Store, Mosaic AI (roadmap) DevOps & CI/CD Azure DevOps / GitHub Actions, Git Scale Numerous log sources • High daily volume in TBs/day raw
Educational Qualifications:
• Required - Bachelor s degree in Computer Science, Information Technology, Computer Engineering or closely related or equivalent. • Preferred - Master s degree in Management Information Systems (MIS), Computer Science, Big Data or Analytics or equivalent.
Travel:
• Open to travel based-up on the nature of the engagement. Thanks & Regards Srikanth Donkani Resource Manager (w): (E): 2260 Haggerty Road, Suite 285 Northville, MI 48167 Equal Employment Opportunity Reliable Software employment does not discriminate on the basis of race, religion, gender, sexual orientation, age or any other basis as covered by federal, state, or local law. Employment decisions are based solely on qualifications, merit and business needs.
