ETL/ELT

Healthcare ETL/ELT Architecture Patterns

Educational design patterns for exploring clinical data ingestion concepts with HL7, FHIR, Kafka, and cloud-native storage.

Disclaimer: This guide is provided by Efneze for educational and technical reference purposes only. HL7®, FHIR®, and related standards are trademarks of Health Level Seven International. Efnezeis not affiliated with or endorsed by Health Level Seven International. For official requirements, always refer to the applicable official specification and your organization's implementation guidance.

The Healthcare Data Pipeline Challenge

Healthcare organizations generate massive volumes of clinical data from dozens of source systems: EHRs, lab information systems, pharmacy systems, imaging, claims, and wearable devices. These systems speak different protocols — HL7 v2 over MLLP, C-CDA via Direct messaging, FHIR over REST, flat file EDI — making integration a core competency for any healthcare data engineering team.

Layer 1: Ingestion

The ingestion layer receives data from source systems and makes it available for downstream processing:

  • HL7 MLLP Listener: Integration engines (Mirth Connect, Rhapsody) listen on TCP ports and receive HL7 v2 messages from hospital systems.
  • FHIR API Polling/Subscriptions: Cloud APIs poll EHR FHIR endpoints or receive FHIR Subscription notifications for new resources.
  • Flat File Drop: Batch EDI files (837, 835, 270/271) are dropped to SFTP or S3 on a schedule.
  • C-CDA Documents: XML documents received via Direct Protocol or API as transition-of-care records. Use a clinical document viewer to inspect source CCDA XML before transformation.

Layer 2: Buffering (Message Queue)

Processing raw messages synchronously is generally better avoided unless explicitly approved by organizational policy. A message queue provides durability, back-pressure, and replay capability:

  • Apache Kafka: Commonly evaluated for higher-volume HL7 stream processing scenarios. Each HL7 message type (ADT, ORU, SIU) maps to a separate topic. Consumers can replay from offset for reprocessing.
  • AWS SQS / Azure Service Bus: Simpler queue options suitable for lower-volume, event-driven architectures.

Layer 3: Transformation (ETL vs ELT)

ETL (Transform First)

  • Parse HL7 → extract fields → map to target schema → load
  • May support strict relational targets
  • Loses original message

ELT (Load Raw First)

  • Dump raw HL7/FHIR to S3 → transform in-place with dbt/Spark
  • Preserves source for replay
  • Commonly evaluated for modern data lakes

Layer 4: Storage

  • Raw Layer (Bronze): Original HL7 messages, C-CDA XML, FHIR JSON stored as-is in S3/ADLS.
  • Normalized Layer (Silver): Parsed, deduplicated, terminology-normalized data in Delta Lake or Iceberg tables.
  • Curated Layer (Gold): Aggregated, patient-timeline views, cohort tables optimized for BI and ML workloads.

Cloud Architecture Options

  • AWS: HealthLake (FHIR), Glue (ETL), S3 (raw), Redshift (warehouse), MSK (Kafka)
  • Azure: Health Data Services (FHIR), Data Factory (ETL), ADLS Gen2, Synapse Analytics, Event Hubs
  • GCP: Cloud Healthcare API (FHIR/HL7/DICOM), Dataflow (streaming), BigQuery (analytics)

Parse HL7 Pipelines

Open HL7 Parser →

Read Engineering Insights

View All Insights →

Related Guides

Streaming Pipelines with Kafka →FHIR vs HL7 v2 →ETL Architecture Deep Dive →

Frequently Asked Questions

What is healthcare ETL?

Healthcare ETL (Extract, Transform, Load) is the process of extracting clinical data from source systems (EHRs, labs, claims systems), transforming it into a standardized format (FHIR, OMOP, custom schemas), and loading it into a target system (data warehouse, data lake, FHIR server) for analytics, reporting, or downstream applications.

What is the difference between ETL and ELT in healthcare?

In healthcare ETL, data is transformed before loading — useful when the target schema is strict (e.g., a relational warehouse). In ELT, raw HL7 or C-CDA data is loaded directly into a data lake as-is, then transformed in-place using SQL or Spark. ELT is commonly evaluated because it preserves original messages for replayability and audit.

What tools are used for healthcare ETL?

Common healthcare ETL tools include Mirth Connect and Rhapsody (HL7 integration engines), Apache Kafka (streaming), Apache Spark and dbt (transformation), Azure Data Factory, AWS Glue, and cloud healthcare APIs like AWS HealthLake and Google Cloud Healthcare API.