Healthcare Data Streaming Concepts with Apache Kafka
Educational architecture concepts for exploring event-streaming patterns involving HL7 ADT examples, lab-result routing examples, FHIR notification patterns, and care-coordination data flows.
Disclaimer: This guide is provided by Efneze for educational and technical reference purposes only. HL7®, FHIR®, and related standards are trademarks of Health Level Seven International. Efnezeis not affiliated with or endorsed by Health Level Seven International. For official requirements, always refer to the applicable official specification and your organization's implementation guidance.
Why Streaming for Healthcare?
Real-time data processing is often explored for time-sensitive use cases. For example, some architectures aim to notify care teams quickly when a patient is admitted, or route time-sensitive operational events for review. A medication administered at bedside may need to update downstream operational or reconciliation systems. Batch-oriented designs may not fit every low-latency workflow.
Apache Kafka is commonly evaluated for event-streaming architectures. Its persistent, replayable log model may be useful in healthcare data environments where auditability and replay need to be considered.
Kafka Topic Design for Healthcare
One educational reference pattern is to organize topics by message or event type, not per patient or facility:
clinical.hl7.adt— All ADT events (admits, discharges, transfers)clinical.hl7.oru— Observation results (lab, vitals)clinical.hl7.orm— Orders (lab, radiology, pharmacy)clinical.hl7.siu— Scheduling events (appointments)clinical.fhir.resources— FHIR resource notificationsclinical.raw.hl7— All raw messages for archival and replay
End-to-End Architecture
An educational reference architecture may look like this:
- Source: EHR (Epic, Cerner) → HL7 v2 over MLLP
- Receiver: Mirth Connect → ACK → Kafka Producer (publishes to raw topic)
- Stream Processor: Kafka Streams or Apache Flink → parse HL7 → route to typed topics
- Consumers: FHIR translator (→ FHIR server), Data warehouse sink (→ Redshift/Snowflake), Alert engine (→ notification services), ML feature pipeline (→ feature store)
- Archival: S3 Sink Connector → persists all messages in raw format for replay
FHIR Subscriptions as Streaming Alternative
FHIR R4B and R5 introduce Topic-Based Subscriptions — a native FHIR mechanism for receiving notifications when resources change. Healthcare organizations can subscribe to a FHIR server and receive WebSocket or REST-hook notifications when a Patient, Observation, or MedicationRequest is created or updated.
FHIR subscription patterns may fit FHIR-native workflows, while Kafka-based patterns may be evaluated for broader event-streaming requirements. The right choice depends on system volume, interoperability needs, governance, and operational constraints.
Key Considerations
- Exactly-once semantics: Kafka transactions and idempotent processing may help reduce duplicate event handling. Medication, billing, and clinical workflows require careful validation by qualified technical, compliance, and operational teams.
- Partitioning strategy: Partition by Patient ID (MRN hash) to ensure all events for a patient are processed in order by the same consumer.
- Retention: Retention settings should be selected based on organizational policy, legal/compliance requirements, operational needs, and architecture constraints. Confirm retention policies with legal, compliance, security, and data governance teams.
- Encryption: Encryption in transit and at rest is commonly considered for sensitive data pipelines. Field-level protection, access controls, and PHI handling should be designed according to organizational security and compliance policies.