Software Engineer - Storage Infrastructure

About GovTech and Role Overview

GovTech leads Singapore's Smart Nation initiatives and drives digital transformation in the public sector. As the Centre of Excellence for Infocomm Technology and Smart Systems, it builds capabilities in Data Science, AI, Application Development, Smart City Tech, Digital Infrastructure, and Cybersecurity. This role offers a meaningful career with continuous learning opportunities aimed at improving lives through technology for public good.

The position focuses on hands-on operation and maintenance of Ceph-based storage services that underlie on-premises Kubernetes and OpenShift platforms. The core responsibilities involve ensuring the reliability and efficiency of production storage services, including monitoring, maintenance, upgrades, capacity management, hardware replacement, troubleshooting, and recovery. This role does not involve developing new distributed storage systems but favors candidates experienced in operating or developing distributed storage platforms.

Key Responsibilities

• Operate and manage Ceph and OpenShift Data Foundation (ODF) storage environments.
• Continuously monitor cluster health metrics such as capacity, latency, throughput, placement groups, and device status.
• Handle OSD and node replacement, recovery, rebalancing, backfill, scrubbing, and scheduled maintenance activities.
• Diagnose storage issues including degraded placement groups, slow operations, quorum failures, device faults, and network-related storage challenges.
• Plan and execute upgrades, expansions, patching, and configuration modifications for Ceph and ODF environments.
• Support block and shared-file storage provisioning through ODF, Ceph CSI, RBD, and CephFS.
• Troubleshoot issues related to persistent-volume provisioning, attachment, mounting, expansion, and performance.
• Maintain operational dashboards, set alerts, develop runbooks, perform capacity planning, and establish recovery procedures.
• Test recovery processes from disk, node, service, and network failures regularly.
• Coordinate infrastructure maintenance involving servers, disks, firmware, and networks with respective teams.
• Participate actively in incident response and root-cause analysis within production environments.

Candidate Requirements

• Proven experience working with distributed storage systems, either in development or operations.
• Strong proficiency in Linux system administration and troubleshooting.
• Deep understanding of distributed storage principles including quorum, data placement, replication, failure domains, recovery, rebalancing, and various storage types (block, file, object).
• Competency in storage monitoring, capacity management, upgrades, expansions, and hardware component replacement.
• Experience benchmarking storage workloads and analyzing performance bottlenecks.
• Ability to troubleshoot complex storage problems spanning software, Linux OS, network, and physical hardware layers.
• Familiarity with hardware aspects such as HDD, SSD, NVMe, Host Bus Adapters (HBA), firmware, and storage networking.
• Experience automating tasks using tools such as Ansible, Python, shell scripting, or equivalent.
• Working knowledge of TCP/IP protocols, bandwidth management, latency, packet loss, and network redundancy strategies.

Employee Benefits and Work Culture

GovTech is committed to creating an inclusive environment that embraces diverse perspectives as a foundation for innovation. Employee benefits follow a total rewards philosophy offering comprehensive perks, wellness programs, and flexible leave policies to support work-life balance. Flexible work arrangements are supported based on role requirements, allowing employees to manage their schedules to achieve their best work.

Learn more about career opportunities and life at GovTech by visiting their official site and social media channels.