Skip to main content
Tallo logoTallo logo

Find Jobs

Find Jobs Near You – Available Work in Your Location

Rainbow bridge illustration

Find your next move.

Your current job is just the starting point. See where workers like you actually went next.

Powered by real career transition data in the U.S. labor market over the past 20 years. How it works

Requires a free account

Skip to job details

Back to Results

Apply for this opportunity

To apply for this job, you'll continue to an external website or email application.

NeatDevs LLC

AI Test Engineer, SME (LLM / Agent Evaluation) - DHS CBP Program

Job Description

AI Test Engineer, SME (LLM / Agent Evaluation)
  • DHS CBP Program 22001 Loudoun County Parkway, Ashburn, VA 20147 Hybrid work $104,000
  • $119,000 a year
  • Full-time Apply with Indeed NeatDevs LLC is a small federal subcontractor with a long-running seat on a U.S. Customs and Border Protection (CBP) program in Ashburn, VA. We are hiring an SME-level AI Test Engineer to own testing of AI components
  • Gemini skills, AI agents, and MCP integrations
  • defining how non-deterministic AI behavior is evaluated and building the automated evaluation and regression suites that run in CI/CD.
Clearance:
U.S. citizenship is required (federal contract). You must be able to obtain a DHS/CBP Public Trust; an active or recent (last 24 months) Public Trust or other clearance shortens your start date but is not required.
Location:
Hybrid
  • on-site in Ashburn, VA roughly one day per week, remote the rest.
You must live in the
DC/MD/VA
area. What you'll do: Define and own test strategy, standards, and quality gates for AI components. Build evaluation approaches for non-deterministic output: golden datasets, scoring rubrics, acceptance thresholds, LLM-as-judge evaluation.
Test agents end to end:
task completion, tool selection and trajectory, guardrails, human-in-the-loop checkpoints.
Test MCP servers and clients:
schema conformance, error handling, authentication, least-privilege access.
Perform adversarial testing:
prompt injection, jailbreaks, data leakage; measure latency, token usage, and cost. Maintain automated suites in CI/CD; track test activity in Jira/Xray; mentor the team on AI testing.
Required:
U.S. citizenship and ability to obtain a DHS/CBP Public Trust; must live in the
DC/MD/VA
area. Bachelor's degree (BA/BS) in Computer Science or related field. 10+ years of software QA and testing experience, including 5+ years in test automation. 2+ years hands-on testing AI/ML or LLM-based applications (chat assistants, RAG, agents, tool integrations). Strong Python test development (pytest); strong API testing (REST, JSON schemas, auth). Working knowledge of agent concepts and the Model Context Protocol (MCP). CI/CD test integration (GitLab, Jenkins).
Nice to have:
Gemini / Vertex AI; Google ADK or LangGraph agent evaluation. LLM eval frameworks (Vertex AI GenAI eval, promptfoo, DeepEval, Ragas).
OWASP LLM
Top 10, NIST AI RMF; Section 508 (ANDI);
ISTQB AI
Testing certification.
Pay and benefits:
$104,000
  • $119,000 per year depending on experience and clearance status.
Full-time W-2 role with health, dental, and vision insurance, life insurance, 401(k), paid time off, and paid federal holidays. NeatDevs LLC is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or protected veteran status.
Pay:
$104,000.00
  • $119,000.
00 per year
Benefits:
401(k) Dental insurance Health insurance Life insurance Paid time off Vision insurance Application Question(s): This position is on a federal contract that requires U.S. citizenship and the ability to obtain a DHS/CBP Public Trust background investigation. Are you a U.S. citizen? (Yes / No) Do you currently hold, or have you held within the last 24 months, a Public Trust or security clearance? If yes, which agency and when? This role is on site in Ashburn, VA about one day per week (remote the rest). Can you commit to that, and do you live in the
DC/MD/VA
area? What is your base salary expectation for this role?
Work Location:
Hybrid remote in Ashburn, VA 20147

Benefits

  • Paid Time Off (PTO)
  • 401(k) Plans
  • Health Insurance
  • Dental Insurance
Review key factors to help you decide if the role fits your goals.
Pay Growth
?
out of 5
Not enough data
Not enough info to score pay or growth
Job Security
?
out of 5
Not enough data
Calculating job security score...
Total Score
99
out of 100
Average of individual scores