Penetration Tester, Frontier AI Evaluation at Cobalt Penetration Tester, Frontier AI Evaluation at Cobalt in Los Altos, California Posted in about 13 hours ago.

Type:

full-time About the role: Cobalt is seeking experienced penetration testers to contribute expert reasoning, technical problems, and evaluation data used to train and assess frontier AI models on security tasks. This opportunity is suited to people who test systems for a living or have done so: penetration testers, red team operators, vulnerability researchers, exploit developers, application security engineers, and serious bug bounty hunters, whether from consultancies, internal security teams, or independent practice. You do not need prior experience in data annotation or AI research. What matters is that you can find and reason about real weaknesses in software and infrastructure unaided, and that you can document how you got there clearly enough for another practitioner to follow. All work is performed against sandboxed environments and purpose-built targets supplied by us or by the lab. We do not accept work performed against systems you are not authorized to test, and we do not accept material obtained without authorization or covered by a client agreement. What you'll do: Depending on the project, you may: Produce written testing traces on security tasks, capturing how you form and test hypotheses, what you rule out and why, and how you arrive at a working approach, rather than only the end result Author novel security problems, capture-the-flag style challenges, and lab environments with verifiable success criteria Evaluate model-generated security content and code, ranking responses, explaining what makes the stronger one stronger, and identifying the specific step at which the technical reasoning breaks down Assess whether stated findings are supported by the underlying evidence, and identify inconsistencies between reported results and what the target actually does Design rubrics and partial-credit criteria for scoring multistep testing and remediation tasks Projects follow their own guidelines, scope rules, and quality standards, and you will work with feedback from reviewers and lab research teams.

Required qualification:

Demonstrable penetration testing experience, evidenced by professional engagements, published vulnerability research or CVEs, a substantive bug bounty record, competitive CTF results, or comparable work Strong hands-on coding ability in at least one of Python, C, C++, Go, Rust, or JavaScript, sufficient to read unfamiliar codebases and write your own tooling rather than only running existing tools Depth in at least one area, for example web and API security, cloud and container security, network and infrastructure testing, mobile security, or binary exploitation and reverse engineering Ability to explain each step of your reasoning clearly in writing, and to produce documentation another practitioner could reproduce Willingness to work strictly within defined scope and authorization, and to sign a confidentiality agreement covering project materials Certifications such as OSCP, OSWE, OSEP, GPEN, or GXPN are useful but not required. Why join

Cobalt AI:

Advance frontier AI where it counts. Apply your testing expertise to data that frontier labs cannot obtain any other way, where your judgment directly shapes how the next generation of models reasons about security. Grow professionally. Expand your influence through evaluation projects, advisory roles, and research collaborations, while developing a working understanding of how frontier models are trained and assessed. Work with a top-tier network. Collaborate with security practitioners and researchers from leading organizations on high-impact, flexible work. Set your own schedule. Flexible 10 to 40 hour weeks that fit around your existing engagements and your life. Competitive pay. Rates vary by project and are determined by a number of factors, including scope, skillset, and experience.