Data/Analytics Lead

Job Overview

We are recruiting for a futuristic technology enterprise focused on developing sovereign, secure, and scalable digital platforms spanning AI, Web3.0, cloud, and health technology domains. You will spearhead the design and delivery of an advanced analytics environment built exclusively using open-source technologies. This role involves constructing large-scale data pipelines, a lake house architecture, and BI infrastructure completely from the ground up while leading a team of data engineers.

Key Responsibilities

• Lead and manage a team of 4-6 data engineers, setting strategic direction and goals.
• Define the analytics platform’s architecture and develop a comprehensive roadmap.
• Develop the data platform using only open-source tools and technologies.
• Establish frameworks for data governance and quality assurance across datasets.
• Design and implement an end-to-end analytics platform based on open-source software.
• Build a data lake house using Cloudian object storage technology.
• Architect ClickHouse OLAP solutions optimized for sub-second query responses.
• Develop ingestion pipelines sourcing from multiple systems including APIs, databases, and SaaS platforms.
• Implement both real-time streaming and batch processing pipelines.
• Create ETL pipelines utilizing Airbyte with CDC, connectors, and data validation.
• Manage workflow orchestration through Apache Airflow featuring DAGs, monitoring, and error handling.
• Employ Kafka for real-time data streaming.
• Optimize data warehousing solutions with ClickHouse features like materialized views and distributed tables.
• Design dimensional data models such as star and snowflake schemas.
• Construct semantic layers to ensure metric consistency.
• Deploy Apache Superset for providing self-service analytics capabilities; optionally integrate Power BI.
• Leverage Redis caching to enhance performance.
• Implement data quality frameworks using Great Expectations and Soda.
• Automate data reconciliation processes and institute lineage, cataloging, and GDPR compliance.
• Develop Python automation tools and custom connectors; apply Continuous Integration/Continuous Deployment (CI/CD) practices for data pipelines.
• Continuously optimize system performance and cost efficiency.

Eligibility Criteria

• Candidates must have a minimum of 6 years of professional experience.
• Applicants should hold a degree in Computer Science Engineering or related fields.

Compensation & Additional Details

• Annual salary ranges between ₹2,500,000 and ₹3,200,000.
• Position is full-time and based onsite in Noida, Uttar Pradesh, India.
• Immediate joining is preferred.
• One vacancy is available.
• Job offer is included as part of the role perks.