Software Engineer, Ingestion & Streaming

About Datavant

Datavant is a leading data collaboration platform dedicated to healthcare, enabling the secure, accessible, and actionable use of health data. We serve a broad range of healthcare organizations, from providers and payers to researchers and life sciences companies. Our mission is to transform healthcare data connectivity and usage to improve patient outcomes and power innovations such as AI in healthcare.

By joining Datavant, you become part of a motivated and collaborative team focused on transformative changes in the healthcare industry.

Role Overview

As a member of the Data & ML Platform organization on the Ingestion & Streaming team based in Galway, you will develop and maintain the core data movement layer of Datavant's platform. This layer includes batch and stream pipelines, change data capture systems, document intake processes, and self-service frameworks for product teams to connect new data sources into Snowflake, Iceberg-based lakehouse, and Databricks. You will contribute by both building direct pipelines where needed and designing scalable frameworks and tooling that allow product teams to onboard data independently and securely.

Key Responsibilities

• Design, develop, and operate data ingestion frameworks that transfer data from operational databases, APIs, document streams, and external feeds into data platforms like Snowflake, Iceberg, and Databricks.
• Maintain and improve the ingestion technology stack including AWS DMS, MWAA/Airflow, Fivetran, and internal tools.
• Develop self-service platforms empowering engineers to independently onboard new sources without deep infrastructure knowledge.
• Create and review Terraform configurations managing AWS network components, IAM roles and policies, compute, and data service resources.
• Collaborate with product and analytics teams to select appropriate ingestion methods such as CDC, batch processing, API integration, and streaming, implementing them end-to-end.
• Take leadership roles in handling production incidents, troubleshooting, and implementing durable solutions to prevent reoccurrence.
• Promote rigorous standards around engineering quality, system observability, cost management, and security in all deliverables.
• Provide guidance and mentorship to mid-level engineers through code reviews, pairing sessions, and design critiques.

Required Qualifications

• Minimum 2 years’ experience in data engineering, platform engineering, or data-centric software development.
• At least 2 years of hands-on use of AWS services, with strong emphasis on networking (VPC, subnets, routing, PrivateLink, security groups), IAM (roles, policies, permission boundaries), and relevant data services.
• 1+ year of production experience writing Terraform or similar Infrastructure-as-Code, including module ownership and managing safe infrastructure deployments.
• 1+ year experience building internal self-service frameworks or platform tooling used by engineering teams.
• Proficient SQL skills with the ability to understand physical data storage in warehouses or lakehouses.
• Hands-on experience with Snowflake or comparable cloud data warehouses and workflow orchestrators such as Airflow or MWAA.
• Familiarity with at least one data ingestion strategy: Change Data Capture tools like AWS DMS or Debezium, managed connectors like Fivetran or Airbyte, or custom pipeline development for API data sources.
• Strong CI/CD practices using Github or equivalent systems including branching, code review, automated testing, and reliable deployment.
• Daily practical use of AI coding assistants such as Claude Code, Cursor, or Copilot, with thoughtful understanding of their impact on team productivity and safeguarding sensitive information (PHI).
• Working knowledge of Python programming; expertise not mandatory.
• Excellent written and verbal communication skills, especially in asynchronous and remote team environments.

Preferred Additional Skills

• Production experience with Iceberg or other open data table formats, especially integrating Snowflake and Databricks.
• Hands-on experience with Databricks or Apache Spark.
• Familiarity with Kubernetes operations.
• Snowflake certification(s).
• Exposure to Microsoft Azure cloud platform.
• Expertise integrating data systems with identity management platforms via SCIM, especially SailPoint.
• Background in healthcare or other regulated sectors such as finance.
• Experience as a DBA, SRE, or DRE managing high-pressure production data systems.

Working at Datavant Ireland

Datavant Ireland is an innovative R&D technology center based in Galway, focused on tackling complex technological challenges with a meaningful purpose. Our work environment is collaborative, diverse, and supportive, prioritizing a culture where every team member thrives and belongs.

We offer competitive and equitable salaries, comprehensive benefits, pension contributions, and flexible hybrid working arrangements.

Equal Opportunity & Accessibility

Datavant is an Equal Employment Opportunity employer committed to diversity and inclusion. We provide reasonable accommodations for applicants with disabilities during the hiring process upon request.