Inference Engineer

Inference Engineer at Acceler8 Talent Inference Engineer at Acceler8 Talent in Los Altos, California Posted in 10 minutes ago.

Type:

full-time

Job Description:

Inference Engineer We're partnered with an AI infrastructure company building next-generation systems for large-scale AI workloads. Their platform is rethinking how inference runs at scale - intelligently orchestrating workloads across heterogeneous hardware to unlock major gains in performance, efficiency, and cost. The team is solving some of the hardest problems in modern AI infrastructure: inference scheduling, KV cache management, runtime optimization, memory efficiency, and low-latency serving across distributed systems. They're looking for engineers who care deeply about how models execute in production - not just training models, but making them fast, scalable, and reliable under real-world load. What You'll Work On Designing and optimizing large-scale inference pipelines Improving latency, throughput, and concurrency under production workloads Building inference runtimes and serving infrastructure Optimizing batching, scheduling, and request orchestration Managing KV cache allocation, reuse, placement, and eviction strategies Improving prefill/decode performance and memory efficiency Profiling bottlenecks across model, runtime, and distributed system layers Collaborating closely with compiler, kernel, and systems engineers What They're Looking For Strong systems engineering fundamentals Experience building or scaling ML inference / model serving systems Deep understanding of performance optimization and memory behavior Experience with runtimes such as vLLM, TensorRT-LLM, or custom serving infrastructure Strong understanding of transformer architectures and attention mechanisms Familiarity with batching, scheduling, concurrency, and cache management Strong Python and/or C++ engineering skills Why Join Work on cutting-edge inference infrastructure and AI systems problems Build systems designed for next-generation AI scale Small, highly technical engineering team Significant ownership and technical impact Opportunity to shape foundational infrastructure for future AI workloads