- Experience
- 6+ yrs
- Salary
- —
- Openings
- 1
- Posted
- 2 days ago
- Work mode
- In office
- Resume
- Required to apply
Where you'll work
Sign in to tell us what does and doesn't work for you here — it sharpens every match we show you.
Job description
About Blend
Blend is a leading AI services company focused on creating significant value for clients by combining human expertise with cutting-edge data science, AI technology, and strategic insight. Our goal is to empower ambitious projects and provide meaningful impacts through innovative data-driven solutions and dedicated professionals.
Role Overview
We seek a seasoned Azure Databricks Engineer proficient in Python, SQL, and Apache Spark to architect, construct, and enhance efficient data pipelines and analytic frameworks on Azure cloud. The successful candidate will have extensive experience handling massive datasets, distributed data processing systems, and modern data engineering methodologies.
Key Responsibilities
- Architect and maintain high-performing, scalable data pipelines utilizing Azure Databricks.
- Develop ETL/ELT workflows leveraging PySpark, Spark SQL, and Python programming.
- Enhance Spark job efficiency focusing on speed, cost-effectiveness, and scalability.
- Process and manage structured and semi-structured data formats such as Parquet, Delta, JSON, and CSV.
- Design and oversee Delta Lake tables ensuring ACID compliance, versioning, and schema flexibility.
- Integrate Azure Databricks with Azure Data Lake Storage Gen2 for streamlined data management.
- Create and optimize complex SQL queries and data transformations.
- Collaborate with interdisciplinary teams including data scientists, analysts, and stakeholders to facilitate analytics and machine learning initiatives.
- Implement rigorous data quality checks, validation procedures, and monitoring protocols.
- Adhere to industry best practices for Azure platform security, access controls, and governance policies.
Qualifications and Skills
- Minimum of 6 years’ experience in data engineering roles.
- Expertise in Azure Databricks platform with hands-on accomplishments.
- Advanced Python skills focused on data manipulation and pipeline construction.
- Deep understanding of SQL including joins, window functions, and performance tuning techniques.
- Proficient experience using Apache Spark and PySpark technologies.
- Skilled in working with Delta Lake architecture and features.
- Familiarity with Azure Data Lake Storage Gen2 management and integration.
- Knowledgeable in distributed computing paradigms.
- Experience in Git for version control and collaboration.
Additional Technical Competencies
- Comfortable with Azure Data Factory for orchestration workflows.
- Experience setting up CI/CD pipelines via Azure DevOps or GitHub Actions.
- Basic understanding of data modeling principles.
- Awareness of cloud security fundamentals and role-based access control within Azure.
- Exposure to real-time streaming technologies like Spark Structured Streaming, Event Hub, and Kafka.