AI Data Engieer

AI Data Engieer AI Data EngineerThe AI Data Engineer and implements data pipelines and retrieval systems for a generativeAI platform. This role is responsible for ingesting, transforming, and indexing domaincontent to enable accurate, grounded responses from AI-powered applications. The AIData Engineer collaborates with agent developers and platform engineers to continuouslyimprove knowledge retrieval quality and coverage.

Key Responsibilities:

Data Engineering & ETL

• Design and develop ETL pipelines for ingesting structured and unstructured datasources into searchable knowledge stores
• Build robust, repeatable ingestion workflows that handle document parsing,transformation, and loading at scale
• Implement data quality checks and validation to ensure accuracy andcompleteness of ingested content
• Utilize AWS services (e.g., S3, Lambda, Step Functions, OpenSearch, Bedrock) tobuild and operate data pipelines and retrieval infrastructureRAG Pipeline Development & Search Tuning
• Architect and optimize retrieval-augmented generation (RAG) pipelines includingdocument chunking strategies, vector embedding generation, and retrievalmechanisms
• Tune search relevance and retrieval quality using vector databases and searchengines, iterating on ranking and filtering approaches
• Evaluate retrieval accuracy using evaluation frameworks and custom benchmarks,establishing measurable quality baselines
• Experiment with embedding models, chunking parameters, and hybrid searchstrategies to continuously improve answer qualityQuality & Testing
• Design and implement test strategies for data pipelines, including validation ofingestion accuracy, data completeness, and transformation correctness
• Develop automated regression tests to detect retrieval quality degradation acrosspipeline changes
• Build and maintain evaluation benchmarks that measure retrieval precision, recall,and relevance
• Champion test-driven development (TDD) practices for pipeline and integrationcodeGenerative AI & Emerging Technologies
• Stay informed of advances in RAG architectures, embedding models, and retrievaloptimization techniques
• Identify opportunities to improve knowledge retrieval through emerging approaches(e.g., contextual retrieval, reranking, hybrid search)
• Collaborate with agent developers to ensure knowledge tools return well-structured, contextually relevant resultsSecurity & Compliance
• Assist with adnce to technology policies and comply with all security controls
• Implement secure coding practices, particularly in handling personally identifiableinformation (PII) and sensitive regulatory data
• Participate in threat modeling and security discussions for API and infrastructurecomponents
• Understand and apply FINRA's security standards and best practices for regulatedfinancial environments#LI-WB