Data & AI Engineer

Must Have Technical / Functional Skills Strong core data engineering competencies with expertise in data concepts and data modelling Experience with Big Data platforms and scalable data pipeline design Hands-on experience with Microsoft Fabric Data Agents and Azure AI Services Knowledge of AI/ML integration within enterprise data platforms Strong analytical and problem-solving capabilities Expertise in performance tuning, monitoring, and optimization Proficiency in PySpark for large-scale data processing Ecommerce domain knowledge with customer behavior analytics experience Experience with Adobe Analytics, Adobe Customer Journey Analytics (CJA), and clickstream data analysis Roles & Responsibilities Experimentation Data Enablement (Silver Layer Ownership) Design, build, and maintain curated Silver-layer datasets in Microsoft Fabric for experimentation reporting and analytics Collaborate with BI/Data Reporting teams to define dimensions, metrics, and joins such as visitor/session, variant, campaign, geo, device, channel, funnel steps, and conversion events Develop reusable and standardized data products including tables and views for dashboards, scorecards, and ad hoc reporting Ensure Silver-layer datasets are clean, conformed, deduplicated, and aligned with agreed business definitions Data Gap Analysis & Assessment Conduct regular gap assessments across experimentation requirements, existing Silver-layer data, and upstream telemetry/source systems Identify missing fields, inconsistent definitions, data latency issues, and join-key mismatches Document business impact, severity, remediation plans, timelines, and dependencies for identified issues Recommend improvements in data models, including facts/dimensions, surrogate keys, grain definition, and conformance rules Gold Layer Requirements & Stakeholder Management Lead workshops with experimentation, BI, measurement, and engineering teams to define Gold-layer reporting requirements Define KPI calculations, attribution rules, scorecard structures, segmentation requirements, governance standards, and refresh SLAs Prepare functional and technical documentation including source-to-target mappings, data dictionaries, validation rules, and acceptance criteria Ensure alignment on single-source-of-truth definitions across CJA, Power BI, and scorecards Data Pipeline Engineering Build and maintain robust pipelines using Microsoft Fabric Pipelines and Azure Data Factory (ADF) Work with 1DS telemetry pipelines or equivalent systems to ensure accurate event and attribute flow into Fabric Implement orchestration, incremental loads, monitoring, and error-handling mechanisms to meet reporting timelines Data Validation & Reconciliation Perform reconciliation between Silver/Gold datasets and Customer Journey Analytics (CJA) Validate event counts, session/user logic, experiment attribution, conversions, and time-window consistency Build automated checks for missing data, duplicate events, schema drift, and metric anomalies Coordinate issue resolution with telemetry, tagging, product engineering, and reporting teams Experimentation Lifecycle Support Ensure datasets are ready for pre-launch checks, measurement, scorecard generation, health checks, and post-test analysis Curate experiment metadata including test IDs, allocation details, start/end dates, KPI metrics, and slicing dimensions Support consistent and reliable experimentation scorecard generation AI Agent Design & Development Design and develop AI-powered agents using Fabric Data Agents, Copilot, and Azure OpenAI Enable automation for scorecard creation, narrative summaries, self-service analytics, anomaly investigation, and metric definition assistance Define agent scope, personas, grounding datasets, RBAC/security models, and evaluation metrics Partner with experimentation and reporting teams for pilot implementation, feedback gathering, and production rollout Documentation, Governance & Operational Excellence Maintain detailed documentation for datasets, transformation logic, metric definitions, pipelines, validation rules, and operational runbooks Establish standards for naming conventions, semantic consistency, versioning, backward compatibility, and performance optimization Provide operational support including monitoring, troubleshooting, incident management, and continuous improvement initiatives