Staff Engineer - Observability Platform

Overview

We invite applications for the role of Staff Engineer focused on designing, building, and managing MongoDB's internal and external Observability platform. This critical system monitors thousands of database clusters for customers, generating actionable alerts to protect essential workloads.

About The Team

The Collections team, part of MongoDB's Observability & Adoption organization, focuses on streamlining telemetry onboarding and collection across the platform. They manage core observability components including onboarding experience, telemetry agents operating on data and control planes, and ingestion services handling metrics, logs, and traces. Their work underpins both internal and customer-facing observability, facilitating insights, alerts, and recommendations.

The team collaborates closely with Data Pipeline, Storage, and Query teams to ensure a robust observability framework. They aim to lower barriers for implementing observability, define telemetry conventions, and build collection systems that exhibit excellent performance, security, documentation, and self-service capabilities.

Role Expectations

Given MongoDB Atlas’s rapid growth, the observability platform handles tens of billions of metric time series and processes petabytes of data including logs and traces. Technologies involved include VictoriaMetrics, Grafana, Splunk, Flink, WarpStream/Kafka, Java, Go, Fluentbit, and OpenTelemetry.

The Staff Engineer will architect and implement high-impact components for scalability, performance, cost efficiency, and reliability. This role entails collaborating with software engineers, product teams, and site reliability engineers to promote best practices for service instrumentation and observability. It is a leadership position within a collaborative culture emphasizing diversity and inclusion.

The position requires physical presence in Cork, with a hybrid working model.

Candidate Profile

• Excited by addressing complex large-scale problems with high standards.
• At least 10 years of experience building, designing, debugging, and tuning distributed or highly concurrent software in Go, Java, C, or C++ for mission-critical systems.
• Proven experience managing latency-sensitive, high-throughput systems.
• Strong foundation in system fundamentals including multithreaded programming, performance profiling, and expert-level coding skills.
• Knowledge of database internals or core data processing system components.
• Experience with indexing or database performance tuning is beneficial.
• Familiarity with observability tools and best practices.
• Excellent verbal and written communication skills with a strong inclination to collaborate and mentor peers.
• Effective time and project management skills with the ability to realistically evaluate project complexity and effort.
• Understanding of information security management principles.

Key Responsibilities

• Lead architecture and development of mission-critical observability components emphasizing scalability, cost efficiency, performance, and resilience.
• Develop improvements that enable engineers and customers to swiftly diagnose root causes of production problems.
• Manage production escalations from technical support and mentor the team in handling such incidents.
• Contribute production-grade database code and foster improvements to code quality within the team.
• Maintain responsibility for core observability codebases ensuring they meet standards for security, availability, durability, and performance.
• Troubleshoot test failures, discover and fix bugs, and safeguard against introducing new bugs.
• Analyze performance implications of code changes to prevent regressions.
• Interview advanced software engineering candidates.
• Continuously update expertise on the latest database and observability research and technologies.
• Lead project management and development for major initiatives company-wide.
• Collaborate across teams and stakeholders on large-scale projects.
• Provide technical guidance regarding product direction, engineering complexity, and project dependencies to product management.
• Work together with engineering leaders to plan product roadmaps.
• Champion robust unit and integration testing to verify correctness.
• Participate in customer support rotations to resolve issues directly.

Success Indicators

• Within a month: grasp MongoDB observability architecture and resolve several bugs.
• Within three months: actively contribute to major upcoming release projects and resolve customer and testing issues.
• Within six months: begin code review responsibilities and evaluation of new feature designs.
• Within a year: lead feature developments, mentor engineers, and influence long-term technical direction of the Observability team.

About MongoDB

MongoDB empowers customers to innovate rapidly with its unified and widely distributed data platform, MongoDB Atlas. Serving over 67,000 customers globally, including 75% of Fortune 100 companies, MongoDB supports modern software and AI-native applications across multiple cloud providers.

The company values employee growth, diversity, inclusion, and wellbeing, offering comprehensive support including affinity groups, fertility assistance, and generous parental leave. MongoDB is committed to equal employment opportunities and accommodation for candidates with disabilities.

Additional Information

This role is based in Cork with hybrid work arrangements. MongoDB is dedicated to fostering an inclusive and diverse workspace.