RedShift Data Engineer

Job Overview

We are looking for a seasoned Redshift Data Engineer possessing over eight years of experience in data engineering, coupled with strong practical knowledge of Amazon Redshift. The successful candidate will play a key role in architecting, building, optimizing, and sustaining scalable data platforms and ETL/ELT workflows within the AWS ecosystem.

Key Responsibilities

• Create and manage scalable data pipelines and ETL/ELT workflows utilizing AWS services.
• Craft, enhance, and optimize intricate SQL queries, stored procedures, views, and data models specifically in Amazon Redshift environments.
• Fine-tune Redshift for performance improvements including query and workload optimization, along with distribution and sort key adjustments.
• Construct and uphold data ingestion workflows from diverse sources into Amazon Redshift.
• Employ AWS Glue, S3, Lambda, Step Functions, and CloudWatch to integrate and orchestrate data efficiently.
• Develop robust data transformation routines using Python and SQL scripting.
• Implement frameworks for ensuring data quality, validation, reconciliation, and ongoing monitoring.
• Design and sustain dimensional data structures including fact and dimension tables.
• Handle substantial datasets, focusing on optimizing processing for both performance efficiency and cost effectiveness.
• Apply incremental loading and Change Data Capture (CDC) methods to data ingestion processes.
• Resolve issues related to pipeline failures, performance bottlenecks, and production incidents promptly.
• Collaborate closely with Data Architects, Business Analysts, Developers, and other involved stakeholders.
• Adhere to established data security, governance protocols, access control policies, and manage AWS IAM effectively.
• Participate actively in code review sessions, technical design meetings, and Agile development workflows.

Mandatory Qualifications and Skills

• Minimum of 8 years' experience in data engineering roles.
• Hands-on expertise with Amazon Redshift is essential.
• High proficiency in advanced SQL coding.
• Strong Python programming and scripting capabilities.
• Experience in constructing and maintaining ETL/ELT pipelines.
• Comprehensive familiarity with AWS cloud services.
• Practical experience using Amazon S3 storage solutions.
• Knowledge of AWS Glue for data cataloging and ETL tasks.
• Experience with data warehousing concepts and dimensional modeling techniques.
• Proven ability in Redshift performance tuning.
• Handling large-scale data processing challenges.
• Comfortable with continuous integration/deployment (CI/CD) processes and version control using Git.
• Good understanding of frameworks for ensuring data quality and governance.

Preferred Additional Skills

• Familiarity with AWS Lambda, Step Functions, EMR, Athena, and CloudWatch monitoring.
• Skillset with Apache Spark and PySpark.
• Experience working with Airflow for workflow orchestration.
• Knowledge of Kafka or streaming data technologies.
• Use of CDC tools such as AWS Database Migration Service (DMS), Debezium, or comparable frameworks.
• Exposure to Terraform or Infrastructure as Code concepts.
• Experience with other cloud data platforms like Databricks or Snowflake.
• Worked within Agile or Scrum development methodologies.

Technical Environment

• Cloud Platforms: AWS, with components such as S3, Glue, Lambda, Step Functions, CloudWatch.
• Databases: Amazon Redshift and PostgreSQL.
• Programming: Python and SQL.
• ETL/ELT Tools: AWS Glue, Airflow, and custom-built Python pipelines.
• Big Data Technologies: Apache Spark and PySpark.
• DevOps: Git, CI/CD pipelines, Terraform.
• Data Modeling Approaches: Star and Snowflake schema, dimensional modeling.
• Streaming: Kafka (desirable).