Pay Growth
?
out of 5
Not enough data
Not enough info to score pay or growth
Senior DataBricks Specialist
Job
IMR Soft LLC
New York, NY (In Person)
Full-Time
Review key factors to help you decide if the role fits your goals.
Job Security
?
out of 5
Not enough data
Calculating job security score...
Total Score
54
out of 100
Average of individual scores
Skill Insights
Compare your current skills to what this opportunity needs—we'll show you what you already have and what could strengthen your application.
Job Description
Senior Databricks Specialist Location:
NYC, NY Duration; 12 Months We're migrating complex on-prem regulatory reporting pipelines from a legacy ETL + Autosys + SQL + Teradata stack to a modern Databricks + Snowflake platform on Azure.The role is hands-on:
design, implement, test, and reconcile production pipelines feeding regulatory reports under strict parity requirements. Must-have Scala / Spark- production experience writing Spark applications in Scala (not just notebooks); comfortable with the DataFrame API, joins, window functions, partitioning, and performance tuning Databricks
- Serverless compute, Unity Catalog, Asset Bundles, Databricks CLI SQL fluency
- confortable writing, analyzing and extracting requirements from complex SQL scripts Snowflake
- schema design, performance, Spark-Snowflake connector Azure
- ADLS, networking basics, secrets/identity (Entra ID / managed identities) Orchestration
- Airflow (DAG authoring, sensors, retries, SLAs) CI/CD
- Artifactory, GitHub Actions pipelines: build, sharded test matrices, artifact promotion through dev → QA → UAT → prod Testing
- Experience in TDD, writing unit tests (ScalaTest, AnyFlatSpec) and BDD (Concordion or equivalent) Data quality & reconciliation
- building automated parity checks against legacy outputs, drift detection, row-level reconciliation tooling Large-scale migrations
- proven track record migrating legacy ETL (Autosys/Informatica/etc.) to cloud data platforms, including dependency mapping and cutover planning Modern data engineering practices
- medallion architecture (Bronze/Silver/Gold), idempotent pipelines, schema evolution, lineage, observability Nice-to-have Financial services / regulatory reporting domain Python (Databricks utilities, tooling) Spec-driven development workflows (specs → plans → tasks → implementation) Gradle (composite builds) and JVM tooling