QA Engineer Storm3 - 2.5 New York, NY Job Details $160,000 - $300,000 a year 11 hours ago Benefits Visa sponsorship Relocation assistance Health insurance Unlimited paid time off Dental insurance Vision insurance 401(k) matching Qualifications Defect resolution root cause analysis Continuous Delivery (CD) implementation Engineering development testing Quality control product testing Process design AI platforms (beyond public GPTs) Quality assurance experience within IT Policy & process development Quality check automation Machine learning (ML) fundamentals DevOps automation Playwright Model evaluation Python Failure analysis Full Job Description Senior QA Lead (AI Systems) About the Company We're a fast-growing Series A AI startup building software for the life sciences industry. Our platform helps enterprise customers create and review complex, regulated content using AI-powered workflows. We're working with leading organizations in the space and are building at the forefront of LLMs, agentic systems, and applied AI. About the Role We're hiring our first dedicated QA Lead to own quality across our AI platform. This is a hands-on engineering role focused on building testing infrastructure, evaluation systems, and automation that ensure both product reliability and AI output quality. You'll be responsible for everything from LLM evaluations and agent testing to Playwright automation, CI/CD quality gates, and release sign-off. The ideal candidate can write production-quality code, build scalable test systems, and think critically about how to measure quality in non-deterministic AI environments. What You'll Do Build evaluation frameworks for LLM and agent-based systems Design regression tests for model, prompt, and workflow changes Create automated testing infrastructure using Python and Playwright Own end-to-end testing across web applications and APIs Develop CI/CD quality gates and automated release workflows Perform manual and exploratory testing on new features Monitor production systems and convert failures into repeatable tests Test agent workflows for failure modes such as tool misuse, looping, hallucinations, and prompt injection Partner with engineering and product teams to establish quality standards Define and scale the QA function as the first dedicated hire Required Experience Strong Python engineering skills Extensive Playwright and test automation experience Experience building and maintaining CI/CD testing pipelines Hands-on manual and exploratory testing expertise Familiarity with AI, ML, or LLM-based systems Experience designing evaluation methodologies and quality metrics Strong debugging, root-cause analysis, and error investigation skills Ability to operate independently and build processes from scratch Preferred Experience Experience with LangChain, LangGraph, or agent frameworks Familiarity with LangSmith, Langfuse, Arize, Braintrust, Promptfoo, OpenAI Evals, or similar tools TypeScript and React experience Adversarial testing or red-team experience Backend experience with async Python services and job queues Experience in regulated industries such as healthcare, pharma, or finance MLOps or LLMOps experience Compensation & Benefits $160,000-$300,000 base salary Meaningful equity package Comprehensive medical, dental, and vision coverage 401(k) with company match Visa sponsorship available Relocation support Unlimited PTO Professional development stipend Location New York City 5 days. Relocation assistance available