Databricks Architect / Admin (W2 only C2H)

Databricks Architect / Admin (W2 only C2H) at

SPECTRAFORCE

Databricks Architect / Admin (W2 only C2H) at

SPECTRAFORCE

in Wethersfield, Connecticut Posted in about 21 hours ago.

Type:

full-time

Job Description:

Position Title:

Databricks Architect / Admin (W2 only C2H)

Employment Type:

Location:

Hartford, CT (Hybrid - 3 Days Onsite / 2 Days Remote)

Schedule:

Monday-Friday, 8:00 AM - 5:00

PM Duration:

6-month

Contract-to-Hire Team Size:

10-12 members (Onshore + Offshore)

Work Environment:

Agile, daily stand-up at 9:00 AM, ServiceNow workflow management

Interview Process:

1-2 rounds (possible third round), panel with hiring manager and team members; technical + behavioral focus, no formal assessment.

Travel:

Minimal (commute only) A senior Databricks architect with deep platform administration experience who can function as a strategic advisor rather than a day-to-day operator. The ideal candidate combines strong Databricks architecture knowledge with AWS infrastructure expertise, cost optimization skills, Linux administration, and enterprise-scale data platform experience. Position Overview The Databricks Architect/Admin is a senior individual contributor responsible for architecting, governing, optimizing, and supporting the enterprise Databricks platform. This role is primarily consultative with selective hands-on responsibilities and serves as the subject matter expert for platform architecture, administration, governance, cost optimization, and enterprise integrations. The environment is already established; the selected candidate will provide strategic guidance, optimization recommendations, and ongoing support rather than building from scratch. This position is a backfill for a retiring resource and requires deep expertise to guide engineering teams and platform direction. The role requires a strong combination of: Databricks architecture expertise Advanced administration knowledge Cloud platform experience (AWS) Unix/Linux administration Cost management and optimization skills Data engineering and integration experience Strategic advisory capability Key Responsibilities Platform Architecture & Design Architect and govern enterprise Databricks environments including: Workspace topology Unity Catalog structures Access control frameworks Define and enforce: Cluster standards Runtime versions Instance pool strategies Auto-scaling policies Design scalable and performant pipelines using: Delta Live Tables (DLT) Databricks Workflows Structured Streaming Define architectural standards for: Delta Lake formats Partitioning Z-ordering

OPTIMIZE/VACUUM

strategies Lead integration architecture with upstream platforms: Fivetran Ab Initio Databricks Consulting & Advisory Serve as the primary Databricks consultant across teams Provide guidance on platform best practices Support technical decision-making Act as the escalation point for Databricks platform issues Mentor teams on platform usage and optimization Recommend new Databricks capabilities and roadmap enhancements Cost Management & Optimization A major focus area during the first 3-6 months.

Responsibilities include:

Own DBU consumption reporting and tracking Identify optimization opportunities across: Interactive clusters Jobs SQL Warehouses Detect inefficiencies such as: Excessive DBU consumption Looping jobs Misconfigured workloads Build cost attribution/chargeback models

Support:

Capacity planning Contract forecasting Vendor discussions Collaborate on monthly cost analyses Unix/Linux Infrastructure & Operations Administer and troubleshoot Linux environments supporting

Databricks Manage:

Init scripts Cluster lifecycle processes Shell scripting Build Bash and Python automation solutions

Support:

Monitoring Log aggregation Maintenance workflows

Manage:

File systems Permissions Data movement operations Perform EC2/VM diagnostics with infrastructure teams Governance, Security & Compliance Design governance frameworks in

Unity Catalog Implement:

Data lineage Tagging Access auditing Partner with cybersecurity teams

Support:

Secrets management Network isolation Encryption Maintain audit documentation and compliance artifacts Automation, AI & Machine Learning Implement automation for: Cluster lifecycle management Scheduling Alerting Self-healing workflows

Leverage:

AutoML MLflow Model Serving Integrate:

Databricks Assistant GitHub Copilot Support scalable feature engineering and ML deployment pipelines Evaluate emerging: AI capabilities Generative AI tools LLM-integrated workflows Required Qualifications 7+ years in data engineering or platform engineering Minimum 4+ years of hands-on Databricks implementation experience Deep expertise in: Unity Catalog Delta Lake Databricks Workflows Delta Live Tables SQL Warehouses Strong Unix/Linux skills: Shell scripting Process management File systems Cron Python and PySpark expertise AWS or cloud platform experience: EC2 Storage Networking IAM/security SQL expertise for reporting and analysis Experience designing scalable and cost-efficient enterprise platforms Experience with automation frameworks Familiarity with AI/ML ecosystem Oracle database experience Git and CI/CD knowledge ServiceNow and Jira exposure Strong communication skills Preferred Qualifications MLflow deployment and model registry experience Generative AI tools: LangChain LlamaIndex RAG workflows Workflow orchestration: Apache Airflow Databricks Workflows Fivetran integration expertise Ab Initio administration or development Data governance platforms: Collibra Alation Experience in regulated industries: Insurance Financial services Terraform/Ansible experience Disaster recovery and resiliency planning Core Technical Skills Databricks Ecosystem Databricks Unity Catalog Delta Lake Delta Live Tables Databricks Workflows SQL Warehouses MLflow AutoML Cloud & Infrastructure

AWS EC2 S3

Networking Private Links Firewalls Programming & Automation Python PySpark Bash/Shell scripting SQL Git Integration Technologies Fivetran Ab Initio Oracle Platforms & Operations ServiceNow Jira Linux/Unix First 3-6 Month Priorities / Challenges Primary focus areas: Cost optimization and Databricks usage tracking Stabilization of Databricks onboarding application (currently MVP) Identification of billing discrepancies and inefficiencies Strategic capacity planning and vendor support Providing platform guidance and consultation to teams Hiring Notes Hartford candidates strongly preferred Exceptional candidates from alternate locations may be considered Budget is below current market expectations Hiring manager noted limited flexibility but open to strong profiles Applicant Notices & Disclaimers For information on benefits, equal opportunity employment, and location-specific applicant notices, click here At SPECTRAFORCE, we are committed to maintaining a workplace that ensures fair compensation and wage transparency in adherence with all applicable state and local laws. This position's starting pay is: $45.00/daily.