Databricks Architect/Admin

Position Title:

Databricks Architect/Admin Work Location:

Hartford, CT (Hybrid)

Assignment Duration:

6 months (possibility of extension)

Job Description:

The Databricks Architect/ADMIN is a senior individual contributor responsible for the design, implementation, and continuous optimization of the enterprise Databricks platform. This role serves as the technical authority for all aspects of the Databricks environment — including workspace governance, Unity Catalog, cluster and compute strategy, data pipeline architecture, and cost management. The Architect works in close partnership with data engineering, analytics, and infrastructure teams, and operates within a broader multi-platform data ecosystem that includes Ab Initio and Fivetran. A strong background in Unix/Linux systems administration and scripting is essential, as the role requires deep engagement with the underlying compute infrastructure supporting the platform. Key Responsibilities Platform Architecture & Design

• Architect and govern the enterprise Databricks environment, including workspace topology, Unity Catalog structure, and access control frameworks.
• Define and enforce standards for cluster configuration, runtime versions, instance pool utilization, and auto-scaling policies.
• Design scalable, performant data pipeline patterns using Delta Live Tables, Databricks Workflows, and structured streaming.
• Establish architectural standards for Delta Lake — including table formats, partitioning strategies, Z-ordering, and

OPTIMIZE/VACUUM

scheduling.

• Lead platform integration design with upstream ingestion tools including Fivetran and Ab Initio, ensuring reliable, governed data delivery. Unix/Linux Infrastructure & Operations
• Administer and troubleshoot Unix/Linux environments underpinning Databricks compute nodes, init scripts, and cluster lifecycle management.
• Develop and maintain shell scripts (Bash) and Python automation for platform operations, monitoring, log aggregation, and maintenance tasks.
• Manage file system operations, permission structures, and data movement tasks in Linux-based storage and compute environments.
• Support EC2/VM-level diagnostics and tuning in coordination with infrastructure and cloud engineering teams. Cost Management & Optimization
• Own DBU consumption tracking and reporting; proactively identify optimization opportunities across jobs, interactive clusters, and SQL warehouses.
• Implement and maintain cost attribution models to support chargeback or showback reporting by team, product, or LOB.
• Partner with the Senior Director on capacity planning, contract utilization forecasting, and multi-year commitment management. Governance, Security & Compliance
• Design and implement data governance frameworks within Unity Catalog, including lineage, tagging, and access auditing.
• Collaborate with Cybersecurity to ensure platform configurations satisfy enterprise security controls, including secrets management, network isolation, and encryption.
• Support audit and compliance activities by maintaining documentation of platform configurations, access policies, and data classification standards. Automation & Artificial Intelligence
• Design and implement end-to-end automation frameworks for platform operations, including cluster lifecycle management, job scheduling, alerting, and self-healing workflows.
• Leverage Databricks AutoML, MLflow, and Model Serving capabilities to support the operationalization of machine learning models within the enterprise data platform.
• Integrate AI-assisted development tooling (e.g., Databricks Assistant, GitHub Copilot) into engineering workflows to accelerate pipeline development and reduce manual effort.
• Identify and drive automation opportunities across ingestion, transformation, data quality, and governance processes — reducing toil and improving platform reliability.
• Collaborate with data science and advanced analytics teams to architect scalable feature engineering pipelines and model deployment patterns on Databricks.
• Evaluate and recommend emerging AI/ML platform capabilities, including generative AI integrations and LLM-backed data workflows, in alignment with enterprise strategy.
• Serve as the primary technical escalation point for Databricks platform issues across data engineering and analytics teams.
• Contribute to sprint planning and project tracking within Jira; manage platform change requests and incidents through ServiceNow.
• Produce and maintain architectural documentation, runbooks, and onboarding materials for platform consumers.
• Evaluate and recommend new Databricks features, partner integrations, and tooling investments in support of the platform roadmap. Required Qualifications
• 7+ years of experience in data engineering or data platform roles, with a minimum of 4 years hands-on Databricks implementation experience.
• Demonstrated expertise with Databricks platform capabilities: Unity Catalog, Delta Lake, Databricks Workflows, Delta Live Ta