Senior Data Engineer / AI Data Platform Engineer

Role Overview

This position requires expertise in developing large-scale data processing systems and orchestrating AI-powered platforms. Candidates should have strong programming abilities with Python or Scala, along with hands-on experience in modern data engineering frameworks and cloud ecosystems.

Key Responsibilities & Skills

• Develop and maintain data pipelines using Apache Spark and Apache Airflow.
• Leverage Databricks on AWS for scalable data processing tasks.
• Design and implement data lake architectures following medallion layering concepts (Bronze, Silver, Gold).
• Build both streaming and batch data workflows integrating REST/GraphQL APIs for seamless data ingestion.
• Manage data governance, cataloging, and monitoring tools to ensure data quality and compliance.
• Implement advanced AI orchestration using RAG (Retrieval-Augmented Generation) and Large Language Model (LLM) frameworks.
• Work with Model Context Protocol (MCP) and utilize AWS Bedrock and AWS Nova Pro services for AI model deployment and management.

Preferred Experience

• Familiarity with media & entertainment or digital platform domains.
• Experience in social media data analytics and sentiment analysis.
• Proficiency with real-time data processing technologies like Kafka or AWS Kinesis.
• Knowledge in MLOps practices and model lifecycle management.