Senior Inference Engineer

Company Overview

TensorX is a sovereign AI infrastructure platform based in Dublin, operating cutting-edge open-weight large language models on proprietary NVIDIA Blackwell GPUs located within European datacenters under strict EU jurisdiction. Our platform provides a drop-in OpenAI-compatible API that ensures complete data privacy — no user inputs are stored after request completion. We serve customers in highly regulated sectors such as finance, healthcare, and government, as well as AI developers and platforms, enabling AI adoption without compromising compliance or performance.

Role Summary

We are seeking a Senior Inference Engineer to join our dynamic engineering team. Reporting directly to the CTO, you will be responsible for managing and optimizing the serving layer interface between our API gateway and inference engines across multiple sites. This includes admission, routing, managing the prefill/decode workload division, KV cache migration over GPU fabric, and autoscaling operations.

Key Responsibilities

• Manage and enhance the existing NVIDIA Dynamo deployment for disaggregated serving, overseeing capacity allocation between prefill and decode workers based on traffic variations.
• Take ownership of routing and admission control mechanisms to ensure requests are directed efficiently to workers with cached prefixes, implementing queueing systems to handle traffic bursts without crashing workers.
• Implement autoscaling strategies targeting latency goals by adjusting GPU allocation per model pool based on traffic patterns and validated parallelism configurations.
• Handle cross-node KV cache transfers utilizing technologies such as NIXL or Mooncake, ensuring network performance is integrated into monitoring and telemetry.
• Operate multi-site serving infrastructures, maintaining independent local serving layers behind a unified access point, and facilitating non-NVIDIA pools via llm-d when necessary.
• Develop and maintain observability frameworks using Prometheus, Grafana, Loki, and GPU telemetry, capturing key model metrics including token latency, queue depths, cache hit rates, and error tracking.
• Coordinate zero-downtime rollouts of serving layer updates with thorough validation protocols shared with GPU Performance teams to prevent unmeasured changes reaching customers.
• Enforce stringent data retention policies ensuring no customer data is retained post-request, through design and operational controls.
• Maintain the serving layer runbook, lead incident response at the fleet level, and document incidents to prevent recurrence.

Required Skills and Experience

• Over 5 years’ professional background in distributed systems, machine learning infrastructure, or production serving environments, with significant experience managing GPU workloads.
• Proven expertise operating large language models in production environments using tools like vLLM, SGLang, or TensorRT-LLM, with in-depth understanding of prefill, decode, KV cache, and batching under load.
• Experience running NVIDIA Dynamo or similar disaggregated serving frameworks such as llm-d.
• Proficient Kubernetes knowledge in production, including deployment techniques, rollouts, custom controllers/operators, and GPU scheduling.
• Strong foundation in distributed systems principles including queueing, backpressure, admission control, routing, and handling retries during traffic bursts.
• Competence in monitoring and telemetry tools (Prometheus, Grafana, Loki) and leveraging metrics for performance optimization.
• Solid understanding of benchmarking methodologies emphasizing controlled variables and reproducible results.
• Proficiency in Python programming and ability to comprehend system code in other languages.
• Regular use of AI-assisted coding tools like Claude Code and Codex to enhance development workflows.
• Effective communication skills, able to clarify technical complexities for both technical teams and non-technical stakeholders, thriving in ambiguous settings.

Desirable Qualifications

• Experience with KV cache transfer and offload systems such as NIXL, Mooncake, or LMCache.
• Expertise in diagnosing GPU fabric performance including RDMA, RoCE, or InfiniBand technologies.
• Familiarity with Gateway API Inference Extension or equivalent Kubernetes-native inference routing solutions.
• Experience managing multi-site or multi-cluster infrastructures.
• Contributions to open-source inference or serving software projects.

Why Join TensorX?

• Work on a live disaggregated serving system actively handling production traffic rather than conceptual roadmaps.
• Operate and influence a fleet of dedicated NVIDIA B300 GPUs owned and operated by TensorX in Dublin and Helsinki.
• Tackle real-world, high-concurrency, bursty, long-context AI workloads that exceed standard benchmark assumptions.
• Be instrumental in architectural decisions shaping the future of our multi-site serving layers and overall inference stack.

Education Requirements

A Bachelor's or Master's degree in Computer Science, Software Engineering, Electrical Engineering, or a related technical field, or demonstrate equivalent practical experience.

Compensation and Benefits

• Attractive remuneration package tailored to experience.
• 25 days of paid annual leave.
• Hybrid work model with the flexibility to work remotely and occasional access to our centrally located Dublin office.
• Complimentary inference tokens for development use.

Additional Information

Note: This is a senior individual contributor role; no agency assistance is needed.