Find Jobs
Find Jobs Near You – Available Work in Your Location
AI Infrastructure Engineer
Job Description
AI Infrastructure Engineer Livermore, CA 94551 Hybrid work From $140,000 a year - Full-time Apply with Indeed This is a hybrid role and in-person work is required in Livermore, CA. Our technical teams rack, power, cool, cable, configure, and troubleshoot the GPU clusters that customers train and serve models on. We are hiring an AI Infrastructure Engineer to be the technical depth of that team. This role spans the full stack of a GPU deployment: you will install and commission the hardware, configure the Linux hosts and the InfiniBand or RoCE fabric that connects them, and own the Tier 2 and Tier 3 tickets that no one else can close. When a node fails repeatedly, you are the person who finds out why. This is a hands-on role. You will still be in the aisle with a torque wrench during deployment windows. You will also write the runbook that prevents the next escalation, tune the fabric that shaved 8% off a customer's job time, and sit in the vendor call explaining why their firmware is the root cause. The work is hybrid. Deployment windows, hardware break-fix, and physical layer work happen on site. Configuration, diagnostics, ticket ownership, automation, and documentation can happen from wherever you work best. What you'll do Escalation and root cause ownership Own Tier 2 and Tier 3 tickets end to end — accept escalations from Tier 1 and support teams, drive them to resolution, and close them with a documented cause rather than a reboot. Perform root cause analysis on server hardware failures: correlate
IPMI/BMC
event logs, MCE and EDAC records, kernel messages, thermal and power telemetry, and vendor diagnostics to distinguish a genuine component failure from firmware, configuration, environmental, or workload causes. Identify recurring failure patterns across the fleet — a DIMM slot that fails repeatedly, a firmware revision correlated with GPU falloff, a rack with marginal airflow — and drive the systemic fix. Lead or contribute to post-incident reviews; translate findings into monitoring, runbook, and standards changes. Serve as the technical escalation point for vendors: build the evidence package, defend the diagnosis, and manage RMA and field-service engagements through to closure. Communicate directly with customers during open incidents and maintenance windows, not only after the fact: status while the issue is live, the notice before a window, and the confirmation after it closes. Concise and factual, with no speculation about cause before it is established. Direct colocation remote hands in writing, to the level of cabinet, rack unit, port, and cable label, and verify the result remotely before accepting the work as done. Participate in a shared on-call rotation for infrastructure incidents. GPU systems and troubleshooting Administer and troubleshoot GPU compute nodes —
NVIDIA H100/H200/B200/B300
SXM and PCIe platforms, AMD Instinct, or comparable accelerators. Diagnose GPU-level faults using vendor tooling: nvidia-smi, DCGM and DCGM diagnostics, XID error interpretation, NVLink and NVSwitch health, ECC and row-remapping events, thermal and power throttling analysis. Install, update, and validate GPU drivers, CUDA toolkits, fabric manager, and container runtimes; manage version compatibility across the fleet. Run and interpret node validation and burn-in — NCCL tests, bandwidth and all-reduce benchmarks, GPU stress tests — before handing capacity to customers. Investigate performance degradation that is not an outright failure: straggler nodes, reduced collective bandwidth, PCIe link width or speed downgrades, NUMA and topology misconfiguration.
Network administration:
AI fabric and tenant connectivity Configure and administer high-performance AI fabrics — InfiniBand (NDR/HDR) and RoCEv2 Ethernet — including switch configuration, firmware management, and topology validation.
Operate InfiniBand subnet management:
subnet manager configuration and failover, partitioning (P_Keys), routing engine selection, and topology verification against the intended fat-tree or rail-optimized design. Troubleshoot fabric health with UFM and the OFED toolchain — ibdiagnet, ibnetdiscover, perfquery, port counters — to isolate symbol errors, link flaps, degraded optics, and bad cables to a specific port.
Tune and validate RoCE deployments:
PFC and ECN configuration, lossless queue setup, and congestion control behavior under collective traffic.
Configure and maintain host-side networking:
RDMA and OFED stacks, NIC and HCA firmware, adapter tuning, bonding, VLANs, and out-of-band management networks. Validate fabric performance after deployment and after change and maintain accurate port maps and topology documentation. Operate the tenant-facing side of the network alongside the AI fabric: VLANs and trunking, LACP port-channels across an MLAG or VLT switch pair, enterprise switch CLI, and the firewall, NAT, VPN, DHCP, and DNS configuration that customer connectivity depends on. Verify tenant segmentation and isolation against the design before capacity is handed over. Linux systems administration Administer the Linux fleet (RHEL, Rocky, or Ubuntu) provisioning, patching, kernel and driver management, storage and filesystem configuration, user and access management. Operate our bare-metal provisioning system (MAAS, Foreman, Ironic, or similar) day to day: commissioning and enrollment, deploy and release cycles, image and cloud-init configuration, netplan and bonded host networking, and storage layout at deploy time, so a new node is production-ready without manual steps. Tune hosts for GPU and HPC workloads: kernel parameters, NUMA and CPU pinning, IRQ affinity, and I/O configuration. Operate monitoring and observability for hosts, GPUs, fabric, and environment; build the alerts and dashboards that catch problems before customers do. Write scripts and tooling (Python, Bash) to automate diagnostics, reporting, and repetitive operational work. Deployment and physical infrastructure Rack, cable, and commission high-density GPU systems, storage, and network equipment; lead the technical side of deployment windows. Install and terminate structured cabling — copper, fiber, DAC and AOC — for 100G to 800G links, with the labeling and routing discipline that fabric troubleshooting depends on.
Plan and verify rack-level power:
PDU selection and placement (including 0U vertical strips), A/B feed redundancy, circuit sizing, and load balancing against measured draw. Contribute to rack and cabinet layout — U-space, weight, airflow and containment, cable management, and power headroom — for new deployments and refreshes. Maintain accurate DCIM and asset records: rack elevations, cable runs, serials, port maps, firmware inventory, and lifecycle state. Write and refine runbooks, install standards, and validation checklists so deployments are repeatable rather than improvised. What you bring 4-7 years combining data center or infrastructure operations with Linux systems administration; time spent in both the aisle and the terminal. Strong Linux administration skills in a production environment — RHEL, Rocky, or Ubuntu — including kernel and driver management, storage, networking, and shell-level diagnostics. Hands-on experience with a bare-metal provisioning system as an operator, not just an installer, including diagnosing nodes that fail to enlist, PXE, or commission. Comfort using Redfish and IPMI as automation interfaces for power, boot order, and firmware, not only as sources of event logs. Demonstrated Tier 2/3 escalation experience: you have been the person a ticket lands on when the first two people could not solve it. Server hardware root cause analysis experience — reading BMC, IPMI, and system event logs and correlating them with kernel and application symptoms to reach a defensible cause. Hands-on experience with high-performance or data center networking: InfiniBand, RoCE, or high-speed Ethernet fabrics, including switch configuration and link-level troubleshooting. Working knowledge of production L2 and L3 networking beyond the AI fabric, enough to isolate whether a customer problem is host, switch, firewall, or upstream. Experience administering or supporting GPU or accelerated compute systems, including driver stacks and vendor diagnostic tooling.
Solid data center fundamentals:
power distribution and redundancy, cooling and airflow, structured cabling, and rack layout. Scripting ability in Python or Bash, and comfort with configuration management such as Ansible. Clear technical writing — incident summaries, runbooks, and RCA document a customer or an executive can read. Experience as the customer-facing technical contact during a live incident, and experience directing remote hands or field technicians in writing at a site you are not standing in. Availability for on-call rotation and occasional after-hours maintenance windows. Nice to have Production experience with large-scale GPU clusters — multi-rack SXM deployments, NVLink/NVSwitch topologies, or rail-optimized fabric designs.
Deep InfiniBand experience:
UFM, subnet manager tuning, ibdiagnet-driven fault isolation, adaptive routing, or SHARP. Familiarity with HPC and AI workload schedulers — Slurm, Kubernetes with GPU operators, or
Run:
ai — and how scheduling interacts with node health. Experience with parallel or high-performance storage (Lustre, GPFS, WEKA, VAST, or NFS over RDMA). Exposure to liquid-cooled and high-density deployments — rear-door heat exchangers, direct-to-chip, CDUs.
Observability stack experience:
Prometheus, Grafana, DCGM exporter, Elasticsearch, or equivalent. Certifications such as RHCE or
RHCSA, NVIDIA
networking or InfiniBand credentials, CompTIA Server+/Network+, or Uptime Institute
DCCA/DCCS.
Experience in a growth-stage environment where standards are being written rather than inherited. Working conditions and physical requirements Part of this role is performed in an operating data center. Candidates should be able to, with or without reasonable accommodation: Lift and maneuver equipment up to 50 lbs unassisted, and heavier items with team or lift assist. Work at heights on ladders and in confined spaces, including under raised floors and in overhead cable trays. Stand, walk, kneel, and reach for extended periods in an environment with sustained noise, cold aisles, and limited natural light. Wear required PPE and follow all electrical and site safety procedures. Compensation and benefits
• Negotiable - Depending on
Experience Pay:
From $140,000.00 per year
Benefits:
401(k) Health insurance
Work Location:
In person
Benefits
- 401(k) Plans
- Health Insurance
- Dental Insurance
