Databricks Solution Expert

Databricks Solution Expert Location US-Remote Job

ID 2026-12115

Overview Planned Systems International, Inc. is hiring a Databricks Solutions Expert that will define platform patterns, build reference implementations, set standards (cluster policies, Unity Catalog governance, CI/CD), and drive enablement so product, analytics, and data science teams can deliver reliable, compliant, and performant insights at scale. Essential Functions and Job Responsibilities Platform Architecture & Design Blueprint the Azure Databricks landing zone: workspace topology (prodon-prod), network architecture (VNet injection, Private Link, NAT), secure connectivity to ADLS Gen2, Azure SQL/MI, Event Hubs, and other data sources. Governance with

Unity Catalog:

multi-region metastores, catalog/schema/table design, data classification tiers, row/column-level security, data lineage, and cross-domain data sharing patterns (Delta Sharing).

Lakehouse foundations:

Delta Lake storage design (bronze/silver/gold), medallion data flow standards, partitioning, Z ordering, OPTIMIZE/VACUUM policies, and performance best practices (Photon, AQE). Scalability for "thousands of teams": multi-workspace strategy, tenancy model (shared vs. dedicated), workspace baselines, cluster policy tiers, and guardrails to prevent noisy-neighbor and cost runaways.

Reliability:

HA/DR strategy, regional deployments, backupestore, versioning, and repeatable environment provisioning via Terraform. Security, Compliance & Access Control Identity & access: Entra ID (Azure AD)

SSO, SCIM

user/group provisioning, service principals/managed identities, attribute-basedole-based access controls mapped to Unity Catalog. Secrets & credentials: Key Vault-backed secret scopes, credential passthrough patterns, token hygiene (PAT governance).

Data protection:

encryption at rest/in transit, private endpoints, data exfil and egress controls, policy-as-code, and audit logging to Log Analytics or secure storage.

Compliance:

enforce enterprise policies (PII/PHI handling), data retention, legal hold, and regulatory reporting with auditable lineage.

Engineering & Enablement Pipelines:

build DLT (Delta Live Tables) pipelines and jobs for batch and streaming (Structured Streaming) with CDC (e.g., via Auto Loader) from enterprise sources.

Performance engineering:

optimize notebooks/SQL/ETL (Photon, caching, skew mitigation), tune cluster sizing, and set standards for reliable, fast jobs.

MLOps:

integrate MLflow for experiment tracking, model registry, feature store (Unity Catalog), and serving patterns w appropriate.

Observability:

end-to-end monitoring (jobs, clusters, UC audits), dashboarding, alerting, and SLOs; integrate with Azure Monitor/Log Analytics.

Enablement:

build reusable reference accelerators (templates, example notebooks, data products), run playbooks, and conduct office hours to uplevel 1000s of teams.

DevOps, Automation & Cost Management Infrastructure-as-Code:

provision workspaces, catalogs, cluster policies, and permissions via Terraform (Databricks provider), with pipelines in Azure DevOps/GitHub.

CI/CD:

notebook/package deployment, testing harnesses (dbx/pytest), environment promotion, and artifact versioning.

FinOps:

cost modeling, budgets/alerts, instance pools, auto-termination, serverless SQL, tagging/chargeback, and usage analytics for executive reporting. Stakeholder Management & Governance Partner with Security, Networking, Compliance, and FinOps to codify enterprise standards. Establish a Lakehouse Platform Council to ratify patterns ...For full information see follow application link. PSI is an Equal Opportunity Employer. All qualified candidates are encouraged to apply, including: Minorities, Women, Individuals with Disabilities, and Protected Veterans.

NOTE TO JOB SERVICE

VEVRAA Federal Contractor requesting priority referrals of Protected Veterans