Senior Software Engineer - Azure Databricks

About CGI

CGI, established in 1976, ranks among the leading independent IT and business consulting firms globally. With a workforce of 94,000 consultants and professionals, CGI offers a comprehensive range of services, including strategic consulting, systems integration, managed IT and business process services, and intellectual property solutions. Operating via a local relationship model supported by a global delivery network, CGI assists clients in digital transformation and accelerating business outcomes. The company's fiscal 2024 revenue reached CA$14.68 billion, and it is publicly traded on the TSX and NYSE.

Position Overview

We are seeking a Senior Software Engineer specializing in Azure Databricks with 8 to 12 years of relevant experience. This is a full-time role based in Bangalore or Chennai working in a hybrid model with shift hours from 1 PM to 10 PM. The role demands passion for programming and the ability to develop scalable, optimized applications primarily focused on data engineering using Azure ecosystem technologies.

Key Responsibilities

• Design, develop, test, deploy, and maintain robust ETL/ELT pipelines leveraging Azure Databricks, PySpark, Spark SQL, Delta Lake, and Azure Data Factory services.
• Implement Lakehouse and Medallion Architecture across Bronze, Silver, and Gold data layers ensuring lineage, replayability, and operational controls are effective.
• Ingest data through advanced techniques such as full and incremental loads, CDC, Auto Loader, schema evolution, watermarking, handling late-arriving data, and restartable processing.
• Create Spark Declarative Pipelines with features like dependency management, data quality assurance, observability, and orchestrate production-grade workflows.
• Design and build Data Vault 2.0 models including hubs, links, satellites, PIT tables, hashes, representing both raw and business vault patterns.
• Develop Kimball dimensional models encompassing facts, dimensions, conformed and slowly changing dimensions, role-playing dimensions, and bridge tables for analytics.
• Construct reusable PySpark and Spark SQL frameworks for data cleansing, standardization, enrichment, deduplication, reconciliation, and business rules.
• Implement enterprise-level data quality practices with DQX framework monitoring completeness, uniqueness, validity, consistency, referential integrity, audit trails, and error handling.
• Develop and manage audit logging, source-to-target reconciliation, control totals, record counting, SLA monitoring, and dashboards with alerts for production support.
• Optimize Spark and Delta Lake workloads by employing partitioning, clustering, caching, broadcast joins, adaptive query execution, skew mitigation, file optimization, and efficient query design.
• Build CI/CD pipelines using Azure DevOps for Databricks code, notebooks, declarative pipelines, Azure Data Factory assets, configurations, and environment promotion.
• Utilize GitHub Copilot judiciously to enhance development velocity, assist in code reviews, generate tests, debug effectively while ensuring secure coding standards.
• Diagnose and troubleshoot production job failures or performance issues, conduct root cause analysis and implement lasting solutions ensuring service restoration within agreed SLAs.
• Collaborate closely with architects, business analysts, QA teams, Power BI developers, and applications teams to deliver data solutions that meet business, quality, security, and reporting standards.

Essential Qualifications

• Hands-on expertise in Databricks, Delta Lake architecture, data modeling techniques, PySpark and Spark SQL development.
• Experience with data quality frameworks such as DQX.
• Proficiency with Azure DevOps CI/CD pipelines deployment and management.
• Ability to build Spark declarative pipelines and familiarity with GitHub Copilot for productive development.
• Strong troubleshooting skills for production job failures and debugging in a complex distributed environment.

Desirable Skills

• Knowledge of Power BI for analytics and reporting.
• Experience with scheduling tools like Autosys.
• Familiarity with Azure Data Factory workflows.

Additional Information

CGI is an equal opportunity employer dedicated to providing inclusive accommodations for applicants with disabilities throughout the recruitment process. Candidates requiring accommodations are encouraged to communicate their needs to ensure an equitable hiring experience.