Tallo logoTallo logo

Astro/Airflow Engineer

Job

Tata Consultancy Services Limited

Pennington, NJ (In Person)

$115,000 Salary, Full-Time

Posted 7 weeks ago (Updated 4 weeks ago) • Actively hiring

Expires 5/27/2026

Apply for this opportunity

This job application is on an outside website. Be sure to review the job posting there to verify it's the same.

Review key factors to help you decide if the role fits your goals.
How is this calculated?
Pay Growth
?
out of 5
Not enough data
Not enough info to score pay or growth
Job Security
?
out of 5
Not enough data
Calculating job security score...
Total Score
100
out of 100
Average of individual scores

Were these scores useful?

Skill Insights

Compare your current skills to what this opportunity needs—we'll show you what you already have and what could strengthen your application.

Job Description

Must Have Technical/Functional Skills
  • 5-8+ years building/operating data or platform systems; 3+ years running Airflow in production at scale (hundreds-thousands of DAGs and high task throughput)
  • Deep Airflow expertise: DAG design and testing, idempotency, deferrable operators/sensors, dynamic task mapping, task groups, datasets, pools/queues, SLAs, retries/backfills, cross-DAG dependencies
  • Strong Kubernetes experience running Airflow and supporting services: Helm, autoscaling, node/pod tuning, topology spread, network policies, PDBs, and blue/green or canary strategies
  • Observability and SRE practices: Prometheus/Grafana/StatsD, centralized logging, alert design, capacity/throughput modeling, performance tuning
  • Security/compliance:
    SSO/OIDC, RBAC, secrets management (Vault/Secrets Manager), auditing, least-privilege connection management, and change control
  • Proven incident leadership, runbook creation, and platform roadmap execution; excellent cross-functional communication
  • Experience operating and leading migrations to/from Airflow
  • OpenLineage/Marquez adoption; Great Expectations or other data quality frameworks; data contracts
  • Cost optimization and capacity planning for schedulers and workers; spot instance strategies
  • Multi-region HA/DR for Airflow metadata DB; backup/restore and disaster drills
  • Building internal developer platforms/portals (e.g., Backstage) for self-service pipelines
  • Contributions to Apache Airflow or provider packages; familiarity with recent AlPs/ Airflow 2.7+ features
  • Architect, deploy, and operate production-grade Airflow on Kubernetes including all components and user application dependencies, with focus on upgrades, capacity planning, HA, security, and performance tuning
  • Operate a multi-scheduler ecosystem: determine when to use Airflow, distributed compute schedulers, or lightweight task runners based on workload requirements; provide unified developer experience across schedulers
  • Build automation infrastructure: Terraform modules and Helm charts with GitOps-driven CI/CD for environment provisioning, upgrades, and zero-downtime rollouts
  • Standardize the developer experience: DAG repo templates, shared operator libraries, connection and secrets management, dependency packaging, code ownership, linting, unit testing, and pre-commit hooks
  • Implement comprehensive observability: metrics collection, dashboards, distributed tracing, SLA/latency monitoring, intelligent alerting, and runbook automation
  • Enable resilient workflow patterns: build idempotency frameworks, retry/backoff strategies, deferrable operators and sensors, dynamic task mapping, and data aware scheduling
  • Ensure reliability at enterprise scale: architect and tune resource allocation (pools, queues, concurrency limits) to support high-throughput workloads; optimize large-scale backfill strategies; develop comprehensive runbooks and lead incident response/postmortems
  • Partner with teams across the organization to provide enablement, documentation, and self-service tooling Salary Range- $110,000-$120,000 a year

Similar remote jobs

Similar jobs in Pennington, NJ

Similar jobs in New Jersey