Data Engineer

Role Overview

This role is critical to ensuring data pipelines do not fail during critical times such as month-end compliance reporting and that data accuracy is maintained end-to-end. You will architect and develop data infrastructure that supports trading analytics, product insights, compliance reporting, and AI/ML systems within a global financial technology organization. Your responsibility includes proactive problem detection and elevating engineering standards across the data team.

Company Mission and Impact

Deriv aims to enable trading opportunities for millions of users worldwide, operating continuously across various regulatory environments. Every trade generates data essential for real-time analytics, compliance verification, fraud detection, and AI-driven customer services. High-quality data engineering here directly affects the integrity of trading prices, compliance reports, and regulatory transparency.

Key Responsibilities

• Design and implement robust ETL/ELT pipelines for both batch and streaming workloads, using AI-enhanced development tools to improve efficiency without sacrificing reliability.
• Integrate observability into pipelines from the outset, including freshness verification, data completeness, schema change alerts, lineage tracing, and automated anomaly detection to ensure continuous governance.
• Proactively identify and rectify data issues prior to their impact on analysts or stakeholders.
• Maintain data contracts encompassing SLAs, schema agreements, and coordination between data producers and consumers, especially related to PII handling, access controls, and regulatory audit requirements.
• Optimize data warehouse performance and cost through efficient querying, partitioning, clustering, and ensuring reliable orchestration.
• Develop scalable data models, using dimensional modeling techniques and semantic layers to enable reuse beyond immediate project needs.
• Continuously identify and resolve gaps in the data platform without awaiting formal assignments.
• Collaborate cross-functionally with analytics, product, finance, and compliance teams to translate business needs into governed, trustworthy data products.
• Conduct code reviews to uphold and raise quality standards and assist in onboarding new engineers by sharing insights and catching errors early.

Who You Are

• A seasoned data engineer with 2 to 8 years of experience focused on creating pipelines that guarantee timely, accurate data delivery beyond just writing SQL queries.
• Expert in cloud-based data ecosystems, particularly Google Cloud Platform (GCP), BigQuery, Airflow, and proficient in Python and SQL, with hands-on experience in pipeline construction for batch and streaming data.
• Experienced with transformation tools such as dbt or Dataform and regularly incorporate AI coding assistants into development workflows.
• Proficient in data modeling approaches like Kimball star schema, Data Vault, and Medallion architecture, applying them selectively based on use case and implementing data contracts and schema registries at scale.
• Capable of diagnosing root causes of data issues, applying systematic fixes, and effectively communicating explanations to both technical and non-technical stakeholders.
• Committed to elevating team skills through knowledge sharing, resource creation, constructive feedback, and assisting teammates in clarifying ambiguous requirements.
• Experience in regulated or compliance-sensitive environments, particularly fintech or trading, including handling personally identifiable information (PII) under audit conditions and optimizing containerized data platforms and warehouse costs.

Technology Stack

• Cloud and Data Warehouse: Google Cloud Platform, BigQuery
• Orchestration Tools: Airflow
• Programming Languages: Python, SQL
• Data Transformation: dbt, Dataform
• Streaming Platforms: Kafka, Google Pub/Sub
• Development Practices: Continuous Integration/Continuous Deployment (CI/CD), version-controlled pipelines, peer code review, AI-assisted development tools

The Challenge and Reward

This position involves complex and demanding responsibilities where pipeline failures can have critical consequences on trading and compliance. You will handle high-pressure situations such as resolving schema mismatches during month-end closures promptly and advocating for stringent governance practices over expedient but fragile solutions. The work you deliver will underpin vital company operations, including dashboards, AI models, and compliance submissions, contributing directly to the company’s success and reliability.