AI Quality Engineer

AI Quality Engineer Luma Financial Technologies - 1.0 Cincinnati, OH Job Details Full-time $90,000 - $115,000 a year 5 hours ago Qualifications AI models Quality control product testing Data validation techniques Test case design AI tools proficiency Technical report writing Financial analysis Prompt engineering Production systems Quality assurance in testing Data interpretation Document review APIs JSON Model evaluation Validation design Full Job Description About Luma Financial Technologies Founded in 2018, Luma Financial Technologies ("Luma") has pioneered a cutting-edge fintech software platform that has been adopted by broker/dealer firms, RIA offices, and private banks around the world. By using Luma, institutional and retail investors have a fully customizable, independent, buy-side technology platform that helps financial teams more efficiently learn about, research, purchase, and manage alternative investments as well as annuities. Luma gives these users the ability to oversee the full, end-to-end process lifecycle by offering a suite of solutions. These include education resources and training materials; creation and pricing of custom structured products; electronic order entry; and post-trade management. By prioritizing transparency and ease of use, Luma is a multi-issuer, multi-wholesaler, and multi-product option that advisors can utilize to best meet their clients' specific portfolio needs. Headquartered in Cincinnati, OH, Luma also has offices in New York, NY, Miami, FL, Zurich, Switzerland and Lisbon, Portugal. For more information, please visit Luma's website. About the role Luma Fintech is building a best-in-class LLM-powered document parsing pipeline that extracts structured data from complex financial product term sheets. We are seeking an AI Quality Engineer to own the daily testing, analysis, and iterative improvement of our Claude API-based extraction system. This role sits at the intersection of financial data operations and applied AI, you will be the person who closes the loop between what the model outputs and what the schema demands. What you'll do Run daily accuracy evaluations against a defined extraction schema, tracking field-level performance across structured product types (autocallables, CLNs, barrier notes, etc.) Design and maintain test cases, regression suites, and gold-standard document sets to benchmark extraction quality over time Diagnose extraction failures, distinguishing between prompt logic issues, schema ambiguity, model hallucinations, and edge-case document formats Iterate on prompt engineering, system instructions, and context design to improve field-level extraction accuracy Work alongside the AI Engineer lead to feed findings into validation logic and rules-based layers that sit on top of LLM output Document failure modes with reproducible examples and root-cause hypotheses Build and maintain evaluation metrics (precision, recall, field coverage, hallucination rate) and report on accuracy trends Flag schema gaps or ambiguities surfaced by real document variance and collaborate with data operations to refine field definitions Contribute to RAG improvements by identifying where retrieved context is insufficient or misleading Qualifications Required Hands-on experience working with LLM APIs (Anthropic, OpenAI, or similar) in a production or near-production context Strong prompt engineering skills, you understand how instruction design affects model behavior, not just output tone Analytical mindset with the ability to systematically isolate variables in model output quality Experience designing structured test cases or evaluation frameworks (QA background is a plus) Familiarity with JSON schema, structured data output, and data validation patterns Ability to read and interpret complex financial or legal documents (term sheets, prospectuses, offering documents), prior financial services exposure strongly preferred Strong written communication; you'll be documenting findings for both technical and non-technical stakeholders Preferred Experience with RAG pipelines and retrieval evaluation Python proficiency for scripting evaluation workflows or parsing outputs Background in structured financial products (autocallables, structured notes, credit-linked notes) Familiarity with evaluation frameworks or tools (e.g., LangSmith, RAGAS, custom evals) What Success Looks Like In 90 days, you have established a repeatable daily evaluation process, a documented baseline of field-level accuracy across product types, and have driven at least one measurable improvement in extraction quality through prompt iteration. Why This Role This is a high-ownership position on a strategic automation initiative with direct visibility to leadership. You won't be maintaining someone else's test suite, you're building the quality layer of a system that processes real financial data at scale. The role will evolve as the system matures, with opportunity to expand into evaluation infrastructure and model improvement strategy. The pay range for this role is: 90,000 - 115,000 USD per year(Cincinnati)