Backend Inference Runtime Engineer Graduate (AML Inference) - 2027 Start

About the Role and Team

Join ByteDance's Data AML team, which powers machine learning training and inference systems integral to products like Douyin, Jinri Toutiao, and Xigua Video. This mid-platform machine learning group supports internal business units by delivering advanced ML computing capabilities and innovating on algorithms to solve business challenges.

As a 2027 graduate hire, you'll engage in pioneering projects, tackle sophisticated technical challenges, and experience expansive professional growth within ByteDance.

Applicants must be able to onboard by the end of 2027; please clearly mention your availability and graduation date on your CV. Candidates can apply for up to two roles across ByteDance and affiliated companies worldwide, with a rolling review process encouraging early application.

Key Responsibilities

• Iterate on and enhance the core architecture of the large model inference engine with a focus on end-to-end GPU performance optimization. Techniques include operator fusion, compilation improvements, GPU memory access refinement, computing pipeline tuning, and asynchronous stream scheduling to overcome inference bottlenecks, improve throughput, and reduce latency.
• Ensure compatibility with diverse GPU and NPU hardware architectures by refining inference engine adaptability and creating a robust, high-efficiency base for large model inference.
• Design, develop, and optimize distributed parallel computing strategies such as tensor parallelism, pipeline parallelism, sequence parallelism, and Mixture of Experts (MoE) parallelism for large model inference. Address challenges related to multi-card deployment, high communication overhead, workload imbalance, and parallel efficiency.
• Stay abreast of cutting-edge technologies in global large model inference, high-performance GPU computing, distributed parallelism, and cache optimization. Benchmark against leading inference frameworks like vLLM and TensorRT-LLM to innovate and continuously improve system performance and cost efficiency, solidifying the team's core technology position.

Candidate Qualifications

Minimum Requirements:

• Completion or imminent completion of a Bachelor's or Master's degree in Computer Science or a related field.
• Strong foundational knowledge in low-level computer science; proficient in C/C++ and Python programming; skilled in CUDA programming; and familiarity with GPU hardware architecture, memory models, compute scheduling, and communication mechanisms.
• Expertise in deep learning operator development and implementation, especially GPU adaptation and optimization for core operators such as matrix calculations, normalization, and activation functions. Capable of independently performing operator reconstruction, memory optimizations, vectorization, and precision adjustments to ensure high performance and stability.
• Understanding of deep learning inference compilation workflows, including computational graph optimization, operator fusion, constant folding, memory reuse, scheduling, and quantization compilation. Able to reduce GPU memory consumption and inference latency via compilation optimizations to significantly boost throughput.
• Proficiency with GPU profiling and analysis tools like Nsight and Profiler to identify bottlenecks in computing, memory access, and scheduling. Possess a mindset geared towards software-hardware co-optimization and the ability to produce and implement systematic performance improvement plans for demanding industrial-scale, low-latency inference workloads.
• Strong communication, collaboration, presentation, and documentation skills. Demonstrates a high level of responsibility, stress management, and the ability to drive complex technical problem resolution and project execution.

Preferred Qualifications:

• Deep understanding of large model inference principles and model parallelism technologies. Experience implementing distributed solutions like tensor parallelism, pipeline and sequence parallelism, along with expertise in multi-GPU communication, load balancing, and parallel efficiency optimization.
• Experience in secondary development and performance tuning of major large model inference frameworks such as vLLM, SGLang, and TensorRT-LLM is highly desirable.

Additional Information

ByteDance, founded in 2012, is dedicated to inspiring creativity and enriching lives globally. Its diverse portfolio includes TikTok, Lemon8, CapCut, Pico, and China-specific platforms including Toutiao, Douyin, and Xigua.

ByteDance values an inclusive workplace that respects diverse skills, experiences, and perspectives, fostering an environment where creativity can thrive for employees worldwide.