Designs and owns LLM inference runtime components, including engine integration, continuous batching, KV cache management, quantized execution, distributed execution, and Kubernetes orchestration. Improves latency, throughput, GPU utilization, and autoscaling across enterprise deployments. Evaluates emerging inference technologies, resolves customer issues, contributes to architecture and code reviews, mentors engineers, and leads engineering practices. Requires expertise in LLM serving, Kubernetes, Go, Python, C++/CUDA, GPU systems, and production model serving. The summary above was generated by AI Senior Inference SW Engineer This role has been designed as 'Hybrid' with a requirement that you will work on average 2 days per week from an HPE office.

Who We Are:

Hewlett Packard Enterprise is the global edge-to-cloud company advancing the way people live and work. We help companies connect, protect, analyze, and act on their data and applications wherever they live, from edge to cloud, so they can turn insights into outcomes at the speed required to thrive in today's complex world. Our culture thrives on finding new and better ways to accelerate what's next. We know varied backgrounds are valued and succeed here. We have the flexibility to manage our work and personal needs. We make bold moves, together, and are a force for good. If you are looking to stretch and grow your career our culture will embrace you. Open up opportunities with HPE. HPE's Private Cloud AI organization is seeking a Senior Software Engineer to build and evolve the model runtime within HPE AI Essentials, the inference platform used by enterprises to operate large language models on infrastructure they own, including air-gapped and sovereign environments. The core engineering challenge in this domain is not model deployment but sustained execution efficiency: achieving low tail latency and high GPU utilization on customer-owned hardware of varying generation and configuration. In this role you will design and implement key components of that runtime

• engine integration, batching, KV cache management, and distributed execution
• together with the Kubernetes orchestration layer that supports it. The primary work location is as listed, but could be any other HPE site location in the US ; however, remote work options will be considered. Responsibilities
• Design, implement, and own major components of the LLM serving deployment, including engine integration, continuous batching, KV cache management and reuse, and quantized execution
• Partner with inference engineering teams and contribute to improving time-to-first-token, inter-token latency, throughput per GPU, and P95/P99 tail latency
• Build and operate distributed execution capabilities, including disaggregated prefill/decode, tensor and pipeline parallelism, and KV cache offload across GPU memory, host memory, and RDMA-attached storage
• Evaluate emerging runtimes, quantization schemes, speculative decoding, and mixture-of-experts serving, and make well-supported recommendations on adoption
• Contribute to the orchestration layer supporting the runtime, including model admission, GPU scheduling and partitioning, cache-aware request routing, and autoscaling
• Triage and resolve customer issues end-to-end, identifying root causes and improving systems and processes to prevent recurrence
• Provide insightful code and design reviews, mentor team members, and lead by example on engineering practices within the team Knowledge and Skills Required
• Familiar with LLM inference engines such as vLLM, SGLang, TensorRT-LLM, TGI, or NVIDIA NIM, including modification of engine internals
• Strong understanding of inference internals, including continuous batching, paged attention, KV cache reuse and prefix caching, chunked prefill, quantization, and speculative decoding
• Working knowledge of tensor and pipeline parallelism, NCCL collective operations, and the GPU memory hierarchy and interconnect characteristics that govern them
• Advanced proficiency in Kubernetes platform architectures, including operators, custom resources, controllers, and scheduling
• Strong programming proficiency in Go and Python, with the ability to read, debug, and profile

C++/CUDA

using tools such as Nsight

• Familiar with debugging/profiling multi-tier application workloads such as RAG, Agents
• Excellent analytical, debugging, and problem-solving abilities Preferred
• Upstream contribution to vLLM, SGLang, TensorRT-LLM, llm-d, LMCache, or KServe
• Disaggregated prefill/decode serving, or KV cache offload and reuse at scale
• RDMA, GPUDirect Storage, InfiniBand, or RoCE
• MIG, fractional GPU allocation, and multi-tenant GPU isolation
• On-premises, air-gapped, or regulated enterprise software delivery Experience and Education
• Minimum of 8 years of experience in Software Engineering, including 1-2+ years working directly on LLM inference runtimes or production model serving
• Degree in Computer Science or related field Accessibility HPE is committed to creating an inclusive and accessible workplace and encourages applications from all qualified individuals, including those with disabilities.

If you believe you require accommodation during any stage of the application or interview process, please submit your request by completing our secure form linked here.

Note:

This option is reserved for applicants needing assistance/reasonable accommodation related to a disability.

What We Can Offer You:

Health & Wellbeing We strive to provide our team members and their loved ones with a comprehensive suite of benefits that supports their physical, financial and emotional wellbeing. Personal & Professional Development We also invest in your career because the better you are, the better we all are. We have specific programs catered to helping you reach any career goals you have — whether you want to become a knowledge expert in your field or apply your skills to another division. Unconditional Inclusion We are unconditionally inclusive in the way we work and celebrate individual uniqueness. We know varied backgrounds are valued and succeed here. We have the flexibility to manage our work and personal needs. We make bold moves, together, and are a force for good.

Let's Stay Connected:

Follow @HPECareers on Instagram to see the latest on people, culture and tech at HPE. #unitedstates

Job:

Engineering Job Level:

TCP_04 "The expected salary/wage range for this position is provided below. Actual offer may vary from this range based upon geographic location, work experience, education/training, and/or skill level.

•

United States of America:

Annual Salary

USD 144,000

• 273,000 in Colorado // 137,000
• 315,000 in North Carolina & Texas The listed salary range reflects base salary.

Variable incentives may also be offered." Information about employee benefits offered in the US can be found at https://myhperewards.com/main/new-hire-enrollment.html The estimated job application period closure is December 1 2026; this timeline is provided for transparency and internal planning purposes. HPE is an Equal Employment Opportunity/ Veterans/Disabled/LGBT employer. We do not discriminate on the basis of race, gender, or any other protected category, and all decisions we make are made on the basis of qualifications, merit, and business need. Our goal is to be one global team that is representative of our customers, in an inclusive environment where we can continue to innovate and grow together.

Please click here:

Equal Employment Opportunity . Hewlett Packard Enterprise is EEO Protected Veteran/ Individual with Disabilities. HPE will comply with all applicable laws related to employer use of arrest and conviction records, including laws requiring employers to consider for employment qualified applicants with criminal histories. Recruitment Fraud Alert We have become aware of an increase in fraudulent recruitment activities in which individuals impersonate our company or authorized recruitment agencies to offer fake employment opportunities. These scams may occur through false websites, emails, social media, or chat-based applications and often aim to obtain personal information or money. Please note that Hewlett Packard Enterprise (HPE), its direct and indirect subsidiaries and affiliated companies, and its authorized recruitment agencies/vendors will never charge a candidate a registration fee, hiring fee, or any other fee in connection with its recruitment and hiring process. We also never request personal information such as back account details, Social Security numbers, or national IDs via social media or chat applications. All legitimate job opportunities will come through official company channels, and candidates are responsible for verifying the credentials of any third party claiming to represent the company. Any reliance on fraudulent communication is at the individual's own risk, and HPE disclaims legal liability for any resulting damages. If you suspect recruitment fraud, do not share personal information or make any payments and report the incident to your local authorities immediately. Read Full Description Hewlett Packard Enterprise Fort Collins, Colorado, USA Office Fort Collins, United States Similar Jobs at Hewlett Packard Enterprise Hewlett Packard Enterprise Principal Services Business Strategy & Planning (Houston, TX) 2 Hours Ago In-Office 124K-285K Annually Senior level 124K-285K Annually Senior level Artificial Intelligence

• Cloud
• Information Technology
• Consulting Leads long-term strategy and business planning for HPE Server Services.

Develops strategic roadmaps, financial and business-driver models, forecasts, KPIs, competitive analyses, and performance reviews. Partners with Product Management, Sales, Finance, and Operations, advises senior executives, and oversees strategic initiatives from planning through execution. Drives resource allocation, operational efficiency, profitability, and alignment across global teams.

Top Skills:

Ai Tools Excel Power BI Hewlett Packard Enterprise Thermal Engineer (Direct Liquid Cooling)

• Houston, TX 2 Hours Ago In-Office 96K-221K Annually Mid level 96K-221K Annually Mid level Artificial Intelligence
• Cloud
• Information Technology
• Consulting Leads mechanical and thermal design for liquid-cooled servers and rack architectures.

Develops cold plates, manifolds, fluid connections, tubing, and serviceability features; performs CFD, thermal analysis, and laboratory testing. Creates qualification plans covering thermal, leak, structural, vibration, materials, corrosion, and manufacturing reliability. Drives root-cause analysis, supplier and manufacturing partner engagements, technical reviews, and design decisions. Mentors engineers while ensuring cooling solutions meet performance, reliability, acoustic, power, and customer requirements.

Top Skills:

3D Cad Computational Fluid Dynamics (Cfd) Coolant Manifolds Cpu/Gpu Cold Plates Creo (Pro/Engineer) Direct Liquid Cooling Quick Disconnects Thermal Interface Materials Thermal Modeling Hewlett Packard Enterprise Category Manager 4 Hours Ago In-Office 121K-284K Annually Expert/Leader 121K-284K Annually Expert/Leader Artificial Intelligence

• Cloud
• Information Technology
• Consulting Owns product category strategy, business planning, pricing, positioning, and sales execution for a significant portfolio or customer segment.

Provides product expertise to sales teams, leads marketing and demand-generation priorities, manages quota responsibility, analyzes market and financial performance, and develops senior relationships with internal teams, customers, partners, and vendors. The role requires advanced IT industry knowledge, complex business planning, negotiation, communication, and cross-functional leadership skills. What you need to know about the Colorado Tech Scene With a business-friendly climate and research universities like CU Boulder and Colorado State, Colorado has made a name for itself as a startup ecosystem. The state boasts a skilled workforce and high quality of life thanks to its affordable housing, vibrant cultural scene and unparalleled opportunities for outdoor recreation. Colorado is also home to the National Renewable Energy Laboratory, helping cement its status as a hub for renewable energy innovation.

Key Facts About Colorado Tech Number of Tech Workers:

260,000; 8.5% of overall workforce (2024 CompTIA survey)

Major Tech Employers:

Lockheed Martin, Century Link, Comcast, BAE Systems, Level 3

Key Industries:

Software , artificial intelligence , aerospace , e-commerce , fintech , healthtech

Funding Landscape:

$4.9 billion in VC funding in 2024 (Pitchbook)

Notable Investors:

Access Venture Partners, Ridgeline Ventures, Techstars, Blackhorn Ventures Research Centers and Universities:

Colorado School of Mines, University of Colorado Boulder, University of Denver, Colorado State University, Mesa Laboratory, Space Science Institute, National Center for Atmospheric Research, National Renewable Energy Laboratory, Gottlieb Institute