Senior Python Data Scraping Engineer

About Mindrift and the Role

Mindrift offers a platform that connects specialists with innovative tech projects, aiming to advance AI technologies by integrating global expertise with state-of-the-art AI development.

We are seeking experienced Senior Python Data Scraping Engineers for the Tendem project, to design and manage advanced web scraping workflows for practical, real-world data extraction applications. This is a part-time, freelance remote position suited for professionals skilled in Python-based data scraping and processing.

Primary Responsibilities

• Design and manage comprehensive data extraction processes for complex websites to produce accurate and structured datasets continuously.
• Utilize both existing tools and bespoke automation approaches to boost efficiency in data gathering, verification, and task management while fulfilling project requirements.
• Adapt techniques to effectively scrape dynamic and interactive web content including JavaScript and AJAX rendered pages.
• Carry out stringent quality control through validation checks, consistency reviews across sources, and ensuring compliance with formatting guidelines before final data delivery.
• Implement scalable scraping methods capable of handling large volumes, monitoring system stability, and adjusting to minor website changes.

Required Qualifications and Experience

• A minimum of five years of relevant professional experience in data engineering, advanced web scraping, automation, or software development is mandatory.
• A Bachelor’s or Master’s degree in Engineering, Computer Science, Applied Mathematics, or similar technical domain is advantageous.
• Strong foundation in scripting and automation, adept at solving complex challenges using modern technologies, with independence and meticulousness in workflow execution.

Technical Expertise

• Proficient Python web scraping skills including libraries like BeautifulSoup and Selenium; familiarity with scraping dynamic and paginated content using proxies.
• Experienced in extracting complex, heterogeneous, and poorly structured data from multiple web hierarchies and formats.
• Competent in data cleaning, normalization, validation, and producing output in formats like CSV, JSON, and Google Sheets.
• Knowledgeable in circumventing anti-bot measures and adjusting for evolving web structures on large datasets.
• Practical familiarity with cloud computing platforms (e.g., AWS) and container technologies such as Docker integrated in scraping pipelines.
• Experience with large language model frameworks including LangChain and OpenRouter to enhance automation implementations.
• Strong command of English at an upper-intermediate level or above is required.

Additional Details

• Ability to operate independently with high attention to data precision and problem-solving is essential.
• GitHub portfolio submission is beneficial but not mandatory.
• Weekly commitment averages 10-20 hours during project phases, though this is contingent on project activity and not fixed.

Compensation

Compensation for this role can reach up to $37 per hour depending on experience and contribution pace. Pay rates may vary for different projects under the platform.