Senior Specialist - Data Engineering

Role Overview

The Senior Data Engineer will be responsible for transforming raw data into actionable, scalable data systems aligned with business objectives. This role requires collaboration with various functional teams to guarantee data availability, reliability, and accessibility to support analytics and strategic decisions.

Main Responsibilities

• Architect, construct, and oversee efficient data pipelines and ETL workflows utilizing technologies such as Apache Spark, Apache Kafka, and Apache Airflow for processing large datasets from multiple origin points.
• Deploy data processing pipelines across diverse platforms including DataFlow (Apache Beam), DataProc (Hadoop/Spark), Data Fusion, and Cloud Composer (Airflow).
• Develop and maintain distributed data systems and storage solutions.
• Partner with data science, marketing, and customer success teams to facilitate data gathering and integrate necessary tools.
• Set up and configure Google Cloud Platform services as part of development and deployment.
• Design and optimize data warehousing and lakehouse solutions like Amazon Redshift, Google BigQuery, AWS S3, and Azure Data Lake Storage, alongside NoSQL databases including MongoDB and Cassandra.
• Collaborate with data architects to apply dimensional modeling techniques (such as star and snowflake schemas) enhancing data analysis capabilities.
• Work with analysts and scientists to fulfill data requirements for advanced analytics and machine learning workflows, including model training and deployment.
• Implement mechanisms for data quality assurance, monitoring, and custom validations to ensure data integrity and completeness.
• Engage in troubleshooting and resolving data issues swiftly to minimize disruptions.
• Remain informed of cutting-edge technologies, especially cloud-native and serverless architectures, contributing to continuous platform improvements.
• Demonstrate expertise in Google BigQuery for proficient data processing.
• Identify and automate repetitive data engineering tasks.
• Provide deployment support to development teams.
• Drive modernization initiatives for data lakes and warehouses.

Qualifications and Skills

• Academic background in Computer Science, Information Technology, or related disciplines; Master’s degree is advantageous.
• Proven experience in data engineering or comparable positions, preferably with Google Cloud Platform proficiency.
• Technical knowledge of data modeling, mining, and segmentation methodologies.
• Advanced skills in programming languages like Python, Java, or Scala; strong command of both SQL and NoSQL databases.
• In-depth comprehension of ETL pipelines, data warehousing principles, and data integration methods.
• Hands-on experience with cloud ecosystems such as AWS, Azure, or Google Cloud Platform and their related services including AWS Glue, Azure Data Factory, and Google BigQuery.
• Excellent analytical and problem-solving aptitude with meticulous attention to detail, adaptable to fast-paced environments and capable of managing competing tasks.
• Effective communication and interpersonal skills fostering productive cooperation with diverse teams and stakeholders.
• Certifications in data engineering cloud platforms (e.g., Google Certified Data Engineer) will be considered a plus.