Senior Software Engineer - Python (LLM Evaluation & Repository Validation)

Position Overview

We are seeking a skilled Senior Software Engineer specializing in Python for a contractor role focused on Large Language Model (LLM) evaluation and repository validation. This is a remote contract for three months requiring a 40-hour workweek with some hours overlapping Pacific Standard Time.

Key Responsibilities

• Assess and analyze GitHub issues found in popular open-source repositories.
• Configure and establish development environments, including Docker setups, for various repositories.
• Modify and execute code locally to test LLM capabilities in debugging and resolving bugs.
• Evaluate the extent and quality of unit testing coverage.
• Engage with active Python repositories to identify engineering assignments suitable for LLM assessments.
• Collaborate with research teams to enhance datasets used in AI model training and evaluation.
• Identify repositories and issues that provide challenging scenarios for LLM troubleshooting.
• Assist in formulating practical software engineering tasks derived from repository histories.

Candidate Requirements

• Proven software development experience with a strong command of Python.
• Familiarity with Git version control, Docker, and setting up development environments.
• Experience handling well-maintained public repositories on GitHub.
• Ability to comprehend, adapt, and test intricate codebases effectively.
• Excellent analytical skills and problem-solving ability.
• Strong proficiency in written English for clear documentation of findings.
• Capability to work autonomously in a fully remote setting.
• Access to a laptop or desktop with a reliable internet connection.

Preferred Experience

• Background contributing to open-source software projects.
• Familiarity with LLM evaluations, artificial intelligence research, or machine learning initiatives.
• Experience with developer tools, automation agents, or continuous integration/continuous deployment setups.

Compensation and Commitment

This assignment is contracted for three months with pay ranging from $19 to $20 per hour. A 40-hour weekly commitment is expected, including some overlap with PST hours.

Application Process

• Submit application form
• Profile review and initial conversation
• Complete assessment tasks
• Final submission and selection