Pay Growth
?
out of 5
Not enough data
Not enough info to score pay or growth
Find Jobs
Find Jobs Near You – Available Work in Your Location
Skip to job details
BS
BrickRed Systems
GenAI/LLM Test Engineer
Choose a Location
This role is available in multiple locations. Pick one to apply.
Review key factors to help you decide if the role fits your goals.
Job Security
?
out of 5
Not enough data
Calculating job security score...
Total Score
73
out of 100
Average of individual scores
Job Description
GenAI/LLM Test Engineer at BrickRed Systems GenAI/LLM Test Engineer at BrickRed Systems in Mercer Island, Washington Posted in about 14 hours ago.
Type:
full-time We are looking for a GenAI/LLM Test Engineer with strong experience in AI/ML product testing, Python, AI evaluation frameworks, SQL, and test automation . The role will focus on testing LLM-based applications, RAG systems, AI agents, prompts, model outputs, and data pipelines.
Required Skills:
LLM/GenAI Product Testing Python & SQL AI Evaluation Frameworks - Ragas, DeepEval, LangSmith, Promptfoo, OpenAI Evals, TruLens Test Automation, Regression & API Testing Performance & Adversarial Testing Generative AI, LLMs & RAG Nice to
Have:
Prompt Engineering / Prompt Testing CI/CD Automation Azure & Snowflake Cloud Data Platform experience
Key Responsibilities:
Design and execute test plans for GenAI/LLM features, prompts, APIs, and applications . Build automated functional, regression, integration, and API test suites . Evaluate model outputs for accuracy, relevance, hallucination, bias, toxicity, and robustness . Develop evaluation frameworks, golden datasets, scoring rubrics, and regression test suites . Perform adversarial/red-team, performance, scalability, and reliability testing . Validate AI data pipelines for data quality, schema, and drift . Integrate automated and AI evaluation testing into CI/CD pipelines . Collaborate with Data Scientists, ML Engineers, and DevOps teams to define quality metrics and acceptance criteria.
Environment:
Primarily Microsoft Azure and Snowflake , with GenAI initiatives leveraging OpenAI and Claude models.
ABOUT BRICKRED SYSTEMS
BrickRed Systems is a global leader in next-generation technology consulting and workforce solutions, specializing in delivering high-quality talent across digital, engineering, marketing, analytics, finance, operations, and business transformation domains . With a strong emphasis on innovation, scalability, and client success , BrickRed Systems helps organizations solve complex business challenges by providing skilled professionals across strategy, technology, creative, and operational functions. BrickRed Systems fosters a culture of continuous learning, collaboration, and excellence , enabling professionals to contribute to high-impact global initiatives while advancing their careers. Through its focus on delivering specialized talent and technology expertise, BrickRed Systems supports organizations in building scalable teams and executing critical business initiatives across diverse industries and functional areas.