AI Application Engineer Scottsdale, AZ, US

• Posted 2 days ago

• Updated 8 hours ago Contract W2 On-site Software Guidance & Assistance Fitment Dice Job Match Score™ 🔢 Crunching numbers... Job Details Skills Orchestration Large Language Models (LLMs) Test Methods Software Development User Experience Use Cases A/B Testing Reasoning Software Development Methodology Web Applications Test Cases Training Regression Testing Dashboard API Workflow Continuous Integration Continuous Delivery Collaboration Documentation Sprint Computer Science Information Systems Software Engineering Business Analysis Agile Test Plans Analytical Skill Data Structure Communication Generative Artificial Intelligence (AI) Prompt Engineering Evaluation Testing Quality Assurance Optimization DevOps Git

POSTMAN JIRA

Microsoft Power BI JSON Microsoft Azure Vector Databases Semantic Search Automated Testing Python SQL JavaScript C# Privacy Artificial Intelligence Recruiting MEAN Stack Customer Service Training And Development

SAP BASIS

Orchestration Large Language Models (LLMs) Test Methods Software Development User Experience Use Cases A/B Testing Reasoning Software Development Methodology Web Applications Test Cases Training Regression Testing Dashboard API Workflow Continuous Integration Continuous Delivery Collaboration Documentation Sprint Computer Science Information Systems Software Engineering Business Analysis Agile Test Plans Analytical Skill Data Structure Communication Generative Artificial Intelligence (AI) Prompt Engineering Evaluation Testing Quality Assurance Optimization DevOps Git

POSTMAN JIRA

Microsoft Power BI JSON Microsoft Azure Vector Databases Semantic Search Automated Testing Python SQL JavaScript C# Privacy Artificial Intelligence Recruiting MEAN Stack Customer Service Training And Development

SAP BASIS

Summary Software Guidance & Assistance, Inc., (SGA), is searching for an AI Application Engineer for a contract assignment with one of our premier Infrastructure Services clients. This role will be remote.

Responsibilities :

Seeking an AI Engineer to help design, test, optimize, and support AI-powered agent experiences across customer and internal business workflows. This role combines prompt engineering, agent orchestration, quality assurance, evaluation framework development, and AI observability. The ideal candidate is curious, analytical, and hands-on, with experience working with Large Language Models (LLMs), prompt design, testing methodologies, and modern software development practices. You will partner closely with product managers, software engineers, architects, and business stakeholders to improve agent behavior, reliability, accuracy, and user experience. Prompt Engineering & Agent Design Design, develop, and iterate on AI agent logic for business use cases. Create, test, and optimize prompts, system instructions, tools, and workflows to improve agent performance. Conduct prompt experiments and A/B testing to measure effectiveness. Analyze agent responses and refine prompting strategies for accuracy, consistency, groundedness, and user satisfaction. Document prompt patterns, reasoning frameworks, reusable templates, and agent design decisions. Collaborate with engineering teams to deploy prompt and agent updates through established SDLC processes. AI Quality Assurance & Testing Design and execute comprehensive test plans covering AI agents, web applications, APIs, and backend services. Develop evaluation frameworks and scoring methodologies for AI output quality.

Create test cases covering:

Accuracy Hallucination detection Safety Tone and style adherence Instruction following Context retention Response consistency Perform regression testing for prompt and model changes. Track, prioritize, validate, and retest defects using Azure DevOps. AI Evaluation & Observability Utilize Arize or similar observability platforms to monitor model and agent performance. Analyze user interactions and evaluation results to identify trends and improvement opportunities. Develop quality metrics and dashboards for: Accuracy Relevance Latency User satisfaction Defect rates Prompt performance Investigate production issues and recommend prompt, workflow, or architectural improvements. Software Engineering & Integration Support integration of AI capabilities into enterprise applications and workflows. Collaborate with developers to validate API integrations, retrieval pipelines, and agent workflows. Assist in CI/CD deployment and release validation activities using Azure DevOps. Create and maintain automated testing assets where appropriate. Documentation & Collaboration Maintain documentation for prompts, evaluation methodologies, testing procedures, and agent configurations. Participate in sprint planning, backlog refinement, and Agile ceremonies. Communicate findings and recommendations to technical and non-technical stakeholders. Share best practices and lessons learned across the team.

Required Skills :

Bachelor's degree in Computer Science, Engineering, Information Systems, or related field. 5-10 years of experience in software engineering, QA engineering, automation testing, business systems analysis, or related technical roles. 1+ years of hands-on experience with Generative AI, LLMs, prompt engineering, or conversational AI solutions. Experience working with Azure DevOps or similar Agile delivery platforms. Experience designing and executing structured test plans. Strong analytical and troubleshooting skills. Experience working with APIs and JSON data structures. Excellent written and verbal communication skills. AI & GenAI Prompt Engineering Agent Design AI Evaluation Frameworks RAG Concepts LLM Testing AI Quality Assurance AI Observability Prompt Optimization Tools Arize AI Azure DevOps Azure OpenAI Git Postman Jira (optional) Power BI (optional) Development Python

SQL REST

APIs JSON Test Automation Frameworks Preferred Skills :

Experience with Arize AI or comparable AI observability tools. Experience with Azure OpenAI, OpenAI, Anthropic, Gemini, or other LLM platforms. Knowledge of Retrieval-Augmented Generation (RAG) architectures. Experience evaluating AI systems using human and automated scoring methods. Familiarity with vector databases and semantic search. Experience creating automated test frameworks. Understanding of responsible AI principles and AI governance. Experience with Python, SQL, JavaScript, or C#. Knowledge of prompt versioning and experimentation practices. By applying for a job with SGA, you agree to allow SGA to process your application for this and future opportunities in accordance with our Privacy Policy. Also, to ensure timely processing, you agree to be contacted by our AI recruiter via email, text, or phone. Message frequency varies and data rates may apply, but you can reply STOP to any SMS message to opt-out of texts and may contact SGA at to opt-out of AI communications. The choice not to engage with AI will not adversely impact your consideration for placement. AI is not used to make any hiring determinations. SGA is a technology and resource solutions provider driven to stand out. We are a women-owned business.

Our mission:

to solve big IT problems with a more personal, boutique approach. Each year, we match consultants like you to more than 1,000 engagements. When we say let's work better together, we mean it. You'll join a diverse team built on these core values: customer service, employee development, and quality and integrity in everything we do. Be yourself, love what you do and find your passion at work. Please find us at . SGA is an Equal Opportunity Employer and does not discriminate on the basis of Race, Color, Sex, Sexual Orientation, Gender Identity, Religion, National Origin, Disability, Veteran Status, Age, Marital Status, Pregnancy, Genetic Information, or Other Legally Protected Status. We are committed to providing access, equal opportunity, and reasonable accommodation for individuals with disabilities in employment, and our services, programs, and activities. Please visit our company to request an accommodation or assistance regarding our policy. #LI-PA1 Employers have access to artificial intelligence language tools ("AI") that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.

Dice Id:

sgainc

Position Id:

26-02241 Posted 2 days ago Company Info About Software Guidance & Assistance Founded in 1981, SGA is a technology and resource solutions provider with a national footprint and headquartered in the shadow of Wall Street. We're a certified women-owned business. We provide contingent staffing, direct placement, and professional and managed services to transform businesses and evolve careers. We're small enough to tailor our services to each client and big enough to deliver for some of the world's largest employers. Our professionals are experts in areas such as IT, finance, accounting, risk, and clinical. SGA provides contingent staffing, direct placement, and professional and managed services nationwide for Fortune 500 companies, mid-size businesses and select startups. Our core skillsets include all areas of technology - business & data analysis, cyber & network security, database administration, development & architecture, infrastructure, program & project management, quality assurance & testing. We also deliver talent across professional business functions such as finance, accounting, risk, and clinical. Our Professional & Managed Services team delivers IT projects through onshore, offshore and hybrid delivery models. We develop software products, modernize applications, add features, and integrate and maintain systems. Our scope covers, among others, complex application suites, data management and visualizations, machine learning and mobile applications.