Staff Machine Learning Inference Engineer

About Cloudera

Cloudera empowers organizations to translate complex data into clear, actionable insights. Managing vast datasets comparable to major hyperscalers, Cloudera serves as the preferred data partner for leading enterprises across various sectors. Fueled by the innovation of the open source community, Cloudera is advancing digital transformation globally.

Role Overview

As adoption of AI increases on our platform, customers deploy models across on-premises, private, sovereign, and public cloud infrastructures. We are looking for a Staff Machine Learning Inference Engineer focused on developing and optimizing production-grade model-serving solutions via Cloudera AI Inference.

Key Responsibilities

• Deploy and fine-tune production model-serving using Cloudera AI Inference for on-premises, private, and public cloud platforms, ensuring compatibility with various models, runtimes, GPUs, and APIs, while maintaining concurrency and reliability.
• Design and implement repeatable benchmarks reflecting realistic input/output lengths, request frequencies, and levels of concurrency. Propose suitable GPU hardware and serving configurations considering model specifications, memory demands, latency targets, throughput, and availability.
• Identify performance bottlenecks throughout serving runtimes, GPU computation and memory, CPU workload, networking, and I/O. Assess optimization strategies including batching, key-value cache handling, parallel execution, quantization, and scaling replicas, validating their effects on throughput and model accuracy.
• Optimize GPU resource use by evaluating inference workloads across different GPU generations and setups, balancing trade-offs between latency, throughput, cost, and operations. Explore GPU sharing or partitioning techniques where beneficial.
• Create benchmarking, profiling, diagnostic, and validation tools to replicate customer scenarios and measure key metrics such as time to generate the first token, inter-token latency, overall latency, throughput, and GPU memory consumption.
• Collaborate with customer teams, platform and applied AI engineers, and product developers to address serving challenges. Convert technical insights into product requirements, optimize system performance, and develop reusable deployment best practices.

Qualifications and Experience

• Eight or more years of engineering experience involving machine learning systems, software development, distributed systems, or performance engineering, showcasing leadership and autonomous ownership.
• Hands-on experience managing GPU-based model-serving workloads in production with at least one inference technology such as NVIDIA NIM, vLLM, SGLang, TensorRT-LLM, Triton, or equivalents.
• Strong analytical skills to clarify ambiguous requirements by forming hypotheses, conducting root cause analysis, and validating solutions through experiments, profiling, and performance measurements.
• Proficient programming skills in Python paired with solid knowledge of Linux, containers, distributed services, and Kubernetes to troubleshoot across the serving stack.
• Experienced in benchmarking and enhancing inference performance, understanding nuances around model weights, runtime overheads, key-value caches, and latency vs. throughput trade-offs.
• Competent in estimating GPU resource allocations and serving setups tailored to workload parameters including context and output lengths, concurrency, traffic spikes, and reliability needs.

Additional Preferred Expertise

• Deploying open-weight large language models or comparable AI models across diverse GPU environments.
• Experience with multi-GPU or multi-node inference coordinating tensor parallelism and distributed workload management.
• Knowledge of GPU sharing/partitioning technologies such as NVIDIA MIG and quantization methods with respect to their impact on model fidelity.
• Analyzing GPU communication bottlenecks using specialized profiling tools like NVIDIA Nsight.
• Developing reusable benchmarks, automation for deployments, or diagnostic utilities suited for enterprise usage including restricted and air-gapped environments.