Azure Databricks & Agentic AI Architect at TalentOla Azure Databricks & Agentic AI Architect at TalentOla in Harwood Heights, Illinois Posted in 12 days ago.

Type:

full-time

Role :

Azure Databricks & Agentic AI Architect Location:

Chicago, IL(Hybrid- 3 days/week in office) Contract role Looking for 16-20 Years minimum experience Mandatory skills

Azure, DataBricks, Agentic AI, ETL, SQL, Data Engineering Role Summary:

We are seeking a visionary Azure Databricks & Agentic AI Architect to design and implement next-generation AI-powered data platforms. This role combines deep expertise in Azure Databricks, Lakehouse Architecture, Data Engineering, and Generative AI to build intelligent, autonomous, and self-optimizing data ecosystems. The ideal candidate will lead the adoption of Agentic AI within Data Engineering and AI-DLC, enabling autonomous data ingestion, transformation, quality management, lineage discovery, observability, optimization, testing, and governance. Key Responsibilities Agentic Data Engineering Leadership

• Design and implement AI-powered Data Engineering platforms leveraging Azure Databricks and Lakehouse architecture.
• Define autonomous workflows using AI Agents for:
• Data ingestion
• Data mapping
• Schema evolution
• Data quality remediation
• Metadata Enrichment
• Pipeline optimization
• Root cause analysis
• Establish frameworks for Human-in-the-Loop (HITL) decision-making and governance. AI-Driven Data Lifecycle (AI-DLC)
• Lead architecture for AI-enabled Data Development Lifecycle across:
• Requirement analysis
• Data modeling
• Pipeline generation
• Automated testing
• Code review
• Documentation
• Deployment
• Monitoring
• Implement AI copilots to accelerate developer productivity.
• Enable automated lineage creation and intelligent impact analysis. Lakehouse & Data Platform Architecture
• Design scalable Lakehouse platforms using:
• Azure Databricks
• Delta Lake
• Unity Catalog
• ADLS Gen2
• Databricks Workflows
• Delta Live Tables Enterprise GenAI Integration
• Architect RAG-based solutions using enterprise data assets.
• Design agent orchestration frameworks using:
• Azure OpenAI
• LangGraph
• Semantic Kernel
• AutoGen
• MCP-enabled architectures
• Build domain-specific AI agents supporting Data Engineering and Analytics teams. AI Governance & Responsible AI
• Define guardrails for enterprise GenAI adoption.

Implement:

• Prompt governance
• Observability
• Cost monitoring
• Auditability
• Explainability
• Security controls
• Establish governance models for autonomous AI agents. AI-Powered Platform Optimization
• Design self-healing data pipelines.

Implement AI-driven:

• Incident triage
• Failure prediction
• Capacity planning
• Cost optimization
• SLA monitoring
• Enable intelligent workload placement and model routing. Technical Skills
• Data Platform
• Azure Databricks
• Delta Lake
• Unity Catalog
• Azure Data Factory AI & Agentic Frameworks
• Azure OpenAI
• Knowledge Graph
• RAG Architecture
• LangChain
• LangGraph
• MCP Protocol
• Vector Databases
• AI Agent Orchestration Data Engineering
• PySpark
• Spark SQL
• Python
• SQL
• ELT/ETL Modernization DevOps & AI-DLC
• Azure DevOps
• GitHub Actions
• CI/CD
• MLOps
• LLMOps
• Evaluation Frameworks
• AI Testing Frameworks Leadership Expectations
• Drive AI-First Data Engineering transformation.
• Define enterprise patterns, accelerators, and reusable AI agents.
• Mentor architects, data engineers, and AI engineers.
• Lead executive conversations on AI adoption, ROI, and transformation roadmaps.