Technical Lead, Spark

About the Role

At Cloudera, we empower enterprises to turn complex data into actionable intelligence. As a leader in data management and analytics, we build scalable, secure, and high-performance data platforms. The Data Platform Pillar is the foundation of our technology, developing core components that enable customers to efficiently store, manage, and process data.

We are seeking a Senior Staff Software Engineer specializing in Spark (Java) with strong expertise in distributed systems. This role focuses on enhancing the Cloudera distribution of Apache Spark and Livy, creating enterprise-grade solutions that operate on thousands of nodes and handle petabytes of data.

Key Responsibilities

• Lead the design and development of new features for the Cloudera data engineering platform, guiding them from prototype to scalable production release.
• Contribute to open source projects including Apache Spark and Livy.
• Develop software using Scala, Java, and Python on modern platforms.
• Build deep knowledge in distributed data processing, including SQL planners, optimization, data layout, and table formats like Apache Parquet and Iceberg.
• Operate and troubleshoot large-scale distributed systems running on hundreds to thousands of nodes.
• Analyze and resolve system deployment issues through root cause investigation and system tests.
• Enhance internal infrastructure to improve performance and reliability.
• Collaborate effectively with cross-functional teams and stakeholders across multiple geographies.

Candidate Profile & Requirements

• Bachelor's or Master's degree in a relevant field or equivalent experience.
• Minimum of 6 years of professional software development experience.
• Proven experience leading and delivering complex product improvements.
• Proficiency in at least one programming language among Java, Scala, or Python, with an eagerness to learn others used by the team.
• Strong background in systems design and software development.
• Commitment to clean code, detail orientation, and delivering quality solutions.
• Excellent verbal and written communication skills.
• Ability to independently research and solve problems with minimal supervision.
• An open mindset, willingness to learn new technologies and build impactful products.
• Experience working with distributed systems.

Preferred Skills

• Familiarity with SQL query planning and optimization.
• Experience using or contributing to Apache Spark or Livy projects.
• Understanding of large-scale distributed systems design, with insights into scaling, performance tuning, and scheduling.
• Experience working with at least one cloud service provider.

Importance of the Role

This position involves tackling complex distributed system challenges by developing foundational software supporting Cloudera Data Platform (CDP) at massive scale. You will enable data scientists and engineers with advanced tools for analytics and modeling, influencing the evolution of enterprise AI and data platforms globally. Collaboration with diverse teams in product, data science, and engineering will drive innovation and establish industry best practices.

Benefits

• Ample paid time off (PTO) policy.
• Dedicated "Unplugged Days" to support work-life balance.
• Flexible work-from-home arrangements.
• Programs promoting mental and physical health.
• Reimbursement for phone and internet expenses.
• Opportunities for ongoing career growth and development.
• Comprehensive benefits with competitive compensation packages.
• Paid time for volunteering activities.
• Access to employee resource groups fostering community and inclusion.

Additional Information

This position is not eligible for immigration sponsorship or relocation assistance.

Equality employer statement: Cloudera is an equal opportunity employer committed to diversity and inclusion.