ISRO - Kafka-based Satellite Data Streaming
Pub/sub messaging system using Apache Kafka for high-throughput satellite data event streaming at ISRO NRSC.
Project Snapshot
Role
Data Engineer Research Intern
Organization
ISRO - National Remote Sensing Centre (NRSC), Hyderabad
Timeline
February – June 2025
Status
Completed
Deployment
Deployed within ISRO NRSC's secure internal infrastructure
Tech Stack
Impact & Results
- High-throughput event streaming for satellite data ingestion pipeline
A pub/sub messaging system using Apache Kafka to handle high-throughput satellite data event streaming. This system forms the backbone of the satellite data archival infrastructure at ISRO NRSC.
The Problem
The satellite data archival system required a robust, scalable event-streaming backbone to handle high-throughput data ingestion, routing, and notification events without bottlenecking.
The Solution
A distributed messaging architecture built on Apache Kafka, allowing various services (ingestion, archival, monitoring, notification) to communicate asynchronously and reliably.
Architecture & Implementation
Event Streaming: Designed and implemented a Kafka-based pub/sub architecture.
Producers & Consumers:
Producers: Ingest satellite data events.
Consumers: Route events to archival, monitoring, and notification services.
Data Used: High-throughput satellite data events and metadata streams.
Tech Stack: Apache Kafka, Python, PostgreSQL.
Challenges & Solutions
High Throughput: Configuring Kafka clusters for extremely high throughput without sacrificing message durability.
Offset Management: Managing consumer offsets across multiple decoupled downstream services.
Deployment: Setting up and tuning distributed systems within a secure internal network.
Key Takeaways
Production-grade Apache Kafka deployment and tuning.
Designing robust pub/sub event-driven architectures.
Scalable stream processing and monitoring.