Pay Growth
?
out of 5
Not enough data
Not enough info to score pay or growth
Find Jobs
Find Jobs Near You – Available Work in Your Location
Find & Apply For Data Engineer Jobs in Cary, North Carolina
Browse jobs from a variety of sources below, sorted with the most recently published, nearest to the top. Click the title to view more information and apply online.
Skip to job details
Now viewing: Lead Data engineer
VI
Vega Intellisoft Inc.
Lead Data engineer
Review key factors to help you decide if the role fits your goals.
Job Security
?
out of 5
Not enough data
Calculating job security score...
Total Score
83
out of 100
Average of individual scores
Job Description
HIRING:
Lead Data Engineer - AI /
Azure Databricks Location:
Cary, NC (On-site / Hybrid)
Employment Type:
Full-Time
Experience:
12-18 Years Any Visa
ABOUT THE ROLE
We are looking for a hands-on Lead Data Engineer to architect and build an enterprise-scale, AI-first Data Hub using Azure Databricks. The role combines advanced data engineering, lakehouse architecture, production-grade AI/LLM solutions, and technical leadership.
KEY RESPONSIBILITIES
Design enterprise lakehouse architecture using Azure Databricks, ADLS Gen2, Delta Lake, and Bronze/Silver/Gold layers. Build metadata-driven ingestion frameworks and scalable ETL pipelines using Python, Scala, and PySpark. Optimize Spark jobs, cluster performance, partitioning, and processing costs. Develop batch, CDC, and streaming pipelines using Azure Data Factory, Event Hubs, Kafka, and Spark Structured Streaming. Build AI-assisted source-to-canonical mapping, automated data quality, anomaly detection, and reconciliation solutions. Design production LLM applications using RAG, agentic workflows, LangChain, LangGraph, or LlamaIndex. Implement semantic layers, knowledge graphs, and conversational data access where applicable. Establish CI/CD, automated testing, governance, security, and data quality standards. Lead architecture reviews, mentor engineers, and contribute hands-on production code.
REQUIRED SKILLS 12-18
years of data engineering or data platform experience. Expert Python, Scala, PySpark, SQL, and Apache Spark. Strong Azure Databricks, Delta Lake, Unity Catalog, and ADLS Gen2. Hands-on Azure Data Factory, Azure Event Hubs, and metadata-driven ingestion. 3+ years building production LLM solutions, including RAG, retrieval, embeddings, and agentic/tool-calling workflows. LangChain, LangGraph, or LlamaIndex with Azure OpenAI, OpenAI, or Databricks Model Serving. Azure DevOps, Terraform, Databricks Asset Bundles, and CI/CD. Strong Azure security, governance, data modelling, and Spark performance tuning. Proven enterprise-scale lakehouse architecture and technical leadership.
PREFERRED SKILLS
Knowledge graphs, text-to-SQL, AI-driven anomaly detection, LLMOps/MLOps, dbt, Great Expectations, and insurance or financial services experience. Interested candidates, please share your resume at: