Staff Engineer - Observability Platform

About the Role

We are recruiting an experienced Staff Engineer to architect, develop, and maintain the observability infrastructure integral to the MongoDB platform. Our Observability stack supports tens of thousands of customers in monitoring their database clusters and generating critical alerts to protect vital workloads.

The Collections team within MongoDB's Observability and Adoption Organization focuses on simplifying telemetry onboarding and data collection across our systems. This team is responsible for key components like onboarding workflows, telemetry agents operating across data and control planes, and ingestion services processing metrics, logs, and traces that facilitate observability for both internal teams and customers.

Our mission is to minimize complexity for teams implementing observability solutions while collaborating closely with product groups to embed best instrumentation practices. We aim to establish clear telemetry conventions and build scalable, secure, performant, and well-documented collection and ingestion systems that enable self-service.

Collaboration spans beyond our immediate team, working with Data Pipeline and Storage & Query teams to maintain an end-to-end stable and scalable observability stack. This position offers a rare chance to influence the monitoring and reliability architecture of one of the most widely used database platforms.

Technical Environment

As MongoDB rapidly grows, the need for high-cardinality observability data has scaled tremendously. The observability infrastructure handles tens of billions of metric time series while processing petabytes of logs, traces, and events. Our technology stack includes VictoriaMetrics, Grafana, Splunk, Flink, WarpStream/Kafka, Java, Golang, Fluentbit, and OpenTelemetry.

Team Culture

We prioritize inclusivity, diversity, and collaboration, fostering a culture where technical leadership thrives and foundational systems are developed with expert-level craftsmanship. This role is based in Cork with a hybrid working arrangement.

Candidate Profile

• Passionate about solving intricate, large-scale problems to a high caliber.
• At least a decade of experience in designing, coding, debugging, and optimizing mission-critical distributed or highly concurrent software in Go, Java, C, or C++.
• Skilled in running low-latency, high-throughput production systems.
• Deep understanding of systems fundamentals such as multithreading, performance profiling, and high proficiency in software engineering.
• Knowledge of database internals or core data processing component development.
• Preferred experience in indexing or database performance optimization.
• Familiar with the observability ecosystem and establish best practices.
• Excellent written and verbal communication skills with a collaborative mindset and mentoring aptitude.
• Strong time and project management abilities with realistic project estimation skills.
• Solid grasp of information security management principles.

Responsibilities

• Design and build scalable, resilient, and cost-effective systems that underpin the mission-critical observability platform.
• Develop solutions that empower MongoDB engineers and customers to swiftly and accurately troubleshoot production issues.
• Lead resolution of escalated production incidents and guide team members in handling these challenges.
• Write and enhance production-quality database code and mentor engineers to elevate code standards.
• Maintain ownership of observability codebases ensuring excellence in security, durability, availability, performance, and maintainability.
• Identify, diagnose, and fix test failures and bugs, while preventing regressions in new code.
• Analyze performance implications of code changes to prevent software regressions.
• Participate in interviewing for senior engineering roles.
• Stay abreast of latest developments in database technology and observability from both industry and academia.
• Lead the development and project management of large-scale company initiatives.
• Collaborate across teams and stakeholders on cross-functional projects.
• Provide technical guidance to Product Management on direction, complexity, and dependencies.
• Shape the product roadmap in partnership with leadership.
• Advocate for robust unit and integration testing to ensure product correctness.
• Support customers and Engineering SREs resolving issues and participate in on-call rotations.

Success Indicators

• Within one month, acquire comprehensive understanding of the observability architecture and resolve multiple bugs.
• Within three months, contribute to major feature development and resolve reported issues.
• Within six months, take on code reviews and evaluate new feature designs.
• Within one year, lead feature development, mentor engineers, and help define the long-term technical vision for the Observability team.

Company Overview

MongoDB is a data platform designed for agility, enabling customers to innovate quickly. As a leader in global, cloud-native database platforms, MongoDB supports over 67,000 customers worldwide and powers many Fortune 100 companies and innovative startups. We aim to cultivate a supportive, inclusive, and enriching workplace, offering employee resource groups, parental leave, fertility assistance, and ongoing personal and professional development opportunities.

Inclusivity and Accessibility

We are committed to equitable hiring and provide accommodations for applicants with disabilities during the recruitment process upon request.