Senior Data Engineer - Specializing in Web Scraping

Overview

A partner company is seeking a Senior Data Engineer focused on web scraping to join their team remotely from the United Arab Emirates. This role involves designing, building, and maintaining advanced web scraping pipelines that generate high-quality datasets to support data-driven decision-making.

Key Duties

• Create, deploy, and maintain web scrapers using various technologies to gather alternative data from multiple web resources.
• Leverage Python and Pandas extensively to clean, transform, explore, and prepare extensive data for further processing.
• Develop robust data pipelines to ingest scraped information into databases and warehousing solutions efficiently.
• Manage scheduled workflows via Apache Airflow and similar orchestration tools to guarantee timely and dependable data availability.
• Collaborate closely with analysts and cross-functional teams to translate data needs into comprehensive technical implementations.
• Implement quality assurance protocols for data accuracy, consistency, and completeness.
• Maintain alert systems, troubleshoot urgent data-related incidents, and resolve operational challenges to ensure seamless daily data operations.
• Design automation tools and applications to enhance web scraping infrastructure capabilities and productivity.
• Contribute to the architecture and design of infrastructure and data products that facilitate data scientists and technical teams.
• Work independently while maintaining collaborative communication with engineering, product, and technology stakeholders to deliver and improve solutions.

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Engineering, or related disciplines.
• Four to six years of experience in data engineering or related technology sectors.
• Proficiency in Python programming and SQL with strong familiarity in database systems.
• Advanced expertise with Python's Pandas library for data cleansing and transformation.
• Hands-on experience with web scraping frameworks and tools like Selenium, Scrapy, Fiddler, Postman, and XPath.
• Extensive knowledge of Apache Airflow for workflow scheduling and pipeline orchestration.
• Good understanding of web technologies such as HTML, JavaScript, and RESTful APIs.
• Experience managing large-scale datasets, including cleaning and transforming data to support analytics.
• Ability to design scalable infrastructure and tools tailored to data engineering needs.
• Excellent communication skills for effective collaboration with technical and non-technical team members.
• Self-driven with strong attention to detail, capable of managing projects independently.
• Preferred: experience with Docker containerization, Kubernetes, Jenkins, GitHub Actions, and AWS services including S3, RDS, SNS, SQS, and Lambda.

Benefits and Work Environment

• This is a fully remote role offering flexibility to work from any location.
• Full-time participation in a collegial, engineering-focused team atmosphere.
• High degree of autonomy and ownership over technical initiatives.
• Opportunities to engage in complex web scraping and data product projects supporting investment analytics.
• Collaborate with diverse teams including engineering, product management, analytics, and data science.
• Be part of a dynamic, innovative technology team dedicated to continuous improvement.
• Professional work setting prioritizing quality, operational excellence, and innovation.

Additional Information

This job posting is managed by a third-party partner responsible for candidate selection and next steps. The selection process is augmented by AI tools for candidate assessment, but final hiring decisions rest with human recruiters.

Applicants should be aware that personal data will be processed per applicable data protection laws to enable recruitment activities and facilitate communication with the hiring employer.