Senior Data Engineer

About Open Innovation AI

Open Innovation AI is a global technology firm specializing in advanced AI workload management solutions. Their primary product, Open Innovation Cluster Manager (OICM), efficiently orchestrates complex AI tasks across multiple infrastructures. The platform is hardware-independent, optimized for a wide variety of GPUs and accelerator hardware, facilitating smooth integration and scalability for enterprise AI applications. Their mission is to simplify and optimize AI workload management, making AI accessible to organizations regardless of size, helping clients reduce operational costs, speed up deployment, and increase return on investment for AI initiatives.

Role Summary

Open Innovation AI is looking for a seasoned Senior Data Engineer to advance development on their MLOps platform and handle deployment of Large Language Models (LLMs). This role requires overseeing scalable data infrastructure as well as contributing significantly to machine learning operations enhancements.

Key Responsibilities

• Create, build, deploy, test, and maintain robust and scalable data systems.
• Participate in the ongoing development and improvement of the company’s MLOps platform.
• Deploy and oversee Large Language Models ensuring scalability and operational effectiveness.
• Design processes for data modeling, mining, and production pipelines.
• Incorporate new data management technologies and engineering tools into existing infrastructure.
• Collaborate with engineering teams and stakeholders to automate machine learning model deployments.
• Propose enhancements to improve the reliability, efficiency, and quality of data handling.
• Stay current with industry trends and integrate relevant innovations into data infrastructure projects.
• Analyze data to spot opportunities for automation and workflow optimization.

Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, or a closely related discipline.
• Practical experience using Apache Airflow and ClickHouse databases.
• Programming proficiency in Python, Java, C++, or Scala.
• Demonstrated background as a Data Engineer with expertise in MLOps and big data technologies.
• Experience deploying and managing Large Language Models.
• Advanced skills in SQL and NoSQL databases and data warehousing strategies.
• Expertise in building data pipelines, data modeling, ETL processes, and workflow management tools.
• Familiarity with cloud platforms such as AWS, Azure, or GCP as applied to data engineering.
• Strong analytical and problem-solving abilities, with a team-oriented mindset.
• Proven project management and organizational skills.