QA Analyst III- #26-10721

$40

• 80 per hour Menlo Park, CA Contract

• Duration:

  12 months
• Remote
• Role Summary
• The Ground Truth Quality Analyst is responsible for maintaining the accuracy and integrity of Meta's AI-powered sales quality governance system.

This system uses LLM-as-Judge technology to assess and evaluate Small Medium Business sales calls daily across 17 quality skills. As part of a 3-person ground truth team, you will conduct independent human reviews of AI-scored call transcripts, verify scoring accuracy, and feed findings back into the system to drive continuous improvement. Your work directly underpins the credibility of automated quality measurement at scale.

• Key Responsibilities
• + Conduct ground truth reviews
• independently evaluate a representative sample of AI-scored Mid-Market sales call transcripts against defined quality skill rubrics. + Support diagnostic deep-dives
• participate in targeted investigations on calls flagged by the governance system for potential scoring issues. + Calibrate regularly
• complete calibration exercises using anchor examples (clear PASS, clear FAIL, borderline) to maintain scoring consistency across the team. + Track inter-rater reliability
• log review outcomes in the governance review tool and flag disagreements with AI scores for resolution. + Categorise disagreements by governance pillar
• classify scoring discrepancies across Score Accuracy, Attribution Integrity, Bias & Fairness, Consistency, Coverage, and Drift to identify systemic patterns. + Feed findings into prompt refinement
• document and communicate patterns in AI scoring errors to support the prompt engineering cycle and improve model performance. + Expand the ground truth dataset
• contribute reviewed and validated call assessments to the growing ground truth corpus used for model benchmarking and governance reporting.
• Required Qualifications
• + 5 years of experience in QA, auditing, or quality review
• ideally in a sales, contact centre, or customer-facing environment. + Demonstrated analytical skills with a strong attention to detail and ability to apply rubrics consistently. + Experience evaluating sales conversations (calls, transcripts, or recordings) against structured quality frameworks. + Comfort working with AI-generated outputs
• ability to critically assess automated scores rather than default to them. + Strong written communication skills for documenting findings and flagging patterns clearly. + Self-directed and reliable in a remote working environment with minimal supervision.
• Preferred Qualifications
• + Experience with LLM evaluation, AI output review, or human-in-the-loop quality processes.

+ Experience in Quality Assurance, performance assessments etc. + Background in sales coaching, enablement, or sales methodology (e.g., MEDDIC, Challenger, SPIN). + Proficiency in multiple languages relevant to EMEA Mid-Market sales. + Experience with inter-rater reliability measurement or calibration frameworks. Pursuant to the California Fair Chance Act, Los Angeles County Fair Chance Ordinance for Employers, Los Angeles Fair Chance Initiative for Hiring Ordinance, and San Francisco Fair Chance Ordinance, qualified applicants will be considered for assignment with arrest and conviction records. Criminal history may have a direct, adverse, and negative relationship with some of the material job duties of this position. These include the duties and responsibilities listed above, as well as the abilities to adhere to company policies, exercise sound judgment, effectively manage stress and work safely and respectfully with others, exhibit trustworthiness, meet client expectations, standards, and accompanying requirements, and safeguard business operations and company reputation.

• About US Tech Solutions:

• US Tech Solutions is a global staff augmentation firm providing a wide range of talent on-demand and total workforce solutions. To know more about US Tech Solutions, please visit www.ustechsolutions.com .
• US Tech Solutions is an Equal Opportunity Employer.
• All qualified applicants will receive consideration for employment without regard to race, colour, religion, sex, sexual orientation, gender identity, national origin, disability, or status as a protected veteran.

• AI Statement:

• By applying, you acknowledge that AI-assisted tools may be used during hiring.
• \#LI-AS132

•