Senior Data Engineer

About Creditsafe

Creditsafe maintains the most extensive privately owned database, supported by a partner network, containing data on 320 million companies across more than 100 countries. As a data-centric organization undergoing significant growth, Creditsafe specializes in aggregating and processing vast volumes of complex data from multiple sources. This enables their clients to gain valuable, actionable insights by filling knowledge gaps, guiding decision making, and uncovering hidden patterns.

Opportunity Overview

This role offers the chance to join a team of highly skilled engineers tasked with designing a new data platform. The platform prioritizes data quality, traceability, and auditability while supporting high throughput and scalability. The infrastructure is based on AWS components, including S3, Iceberg, Aurora, DynamoDB, Lambda, Fargate, ECS, Airflow, and Spark.

The platform aims to manage over 100 million data objects with frequent daily updates, ensuring additions, deletions, and corrections are auditable. A complex matching algorithm will assign new updates to existing records. The processing stack centers on Python applications that transform raw data into API-ready formats efficiently. Additionally, the team develops highly available, low-latency APIs to deliver data rapidly to clients.

Key Responsibilities

• Contribute actively to the codebase and engage in peer code reviews.
• Architect and develop a metadata-driven, event-based distributed data processing system using Python, Spark, Airflow, Iceberg, DynamoDB, AWS Glue, and S3.
• Lead the building and scaling of Creditsafe's data pipelines combining batch and streaming methods, targeting latency under 100ms.
• Analyze domain-specific data to offer recommendations for product enhancements.
• Utilize AI-assisted tools such as Perplexity, Cursor, and CoPilot to support various aspects of development.

Candidate Profile

• Deep understanding and application of best practices in data engineering.
• Proficient in producing clean, high-performance code integrated with cloud environments.
• Proven commercial experience developing production-grade Python data pipelines.
• Strong enthusiasm for continuous skill development through technical challenges, problem solving, and collaborative whiteboarding sessions.
• Excellent communicator capable of articulating ideas clearly and receptive to others' perspectives.
• Experienced in mentoring junior engineers and resolving complex technical issues with a broad knowledge base.
• Active collaboration through documentation sharing and transparent decision-making.
• Ownership and accountability for delivering end-to-end solutions from design through to production deployment.

Why Join Creditsafe?

• Defined career paths facilitating growth as a technical leader or in related disciplines.
• Access to extensive learning resources, regular training, and designated focus periods for skill enhancement.
• Promotion of work-life balance through flexible working hours and options for hybrid work models.
• Participation in innovative projects featuring the latest cloud computing and data management technologies.

Technology Stack

Python, Linux, Airflow, AWS DynamoDB, S3, Glue, Athena, Redshift, Lambda, API Gateway, Terraform, CI/CD.

Employee Benefits

• Competitive salary with bonus schemes.
• 25 days annual leave plus public holidays.
• Hybrid working arrangements supported.
• Healthcare and company pension schemes.
• Cycle to work program and wellbeing initiatives.
• Global company gatherings and events.
• Extensive e-learning resources and clear career advancement opportunities.