ISRO Satellite Data Archival, Monitoring & Notification System
Secure, scalable system for archiving petabyte-scale satellite data at ISRO NRSC - with real-time monitoring, event streaming via Kafka, and automated notifications.
Project Snapshot
Role
Data Engineer Research Intern
Organization
ISRO - National Remote Sensing Centre (NRSC), Hyderabad
Timeline
February – June 2025
Status
Completed
Team Size
2 (co-developed with Arushima; mentored by ISRO scientists)
Deployment
Deployed within ISRO NRSC's secure internal infrastructure
Tech Stack
Impact & Results
- Handles terabytes of national satellite data
- Near real-time event monitoring and alerting
- Research paper co-authored (pending formal submission)
- ISRO Letter of Recommendation issued by C. Pradeep, Scientist/Engineer, NRSC
- ISRO Project Completion Certificate issued by Dr. G. Prasad, Group Head ICIG/DPA, NRSC
An automated system built during a research internship at ISRO NRSC for managing, monitoring, and notifying on satellite data stored in petabyte-scale object storage (ActivScale). This system handles massive datasets with near real-time event streaming via Kafka.
The Problem
ISRO generates massive volumes of satellite data continuously. Managing, monitoring, and notifying stakeholders about this data stored in petabyte-scale object storage (ActivScale) was a manual and inefficient process.
The Solution
A scalable data engineering pipeline and monitoring system that automates the archival of massive satellite datasets, monitors ingestion health in near real-time, and notifies scientists and operators on key events using Kafka-based pub/sub.
Architecture & Implementation
Archival: Automated system for satellite data archival into ActivScale object storage.
Monitoring Layer: Tracks data ingestion status, storage health, and archive completeness.
Event Streaming: Kafka-based pub/sub messaging for high-throughput data event streaming (Producers: ingestion services; Consumers: archival and monitoring services).
Notification System: Alerts scientists and operators on events like new data arrival, archival completion, and anomalies.
Data Used: Petabyte-scale satellite imagery and metadata.
Tech Stack: Python, Apache Kafka, ActivScale, AWS S3, PostgreSQL.
Challenges & Solutions
Scale: Handling petabyte-scale data without overwhelming the network infrastructure.
Reliability: Designing a fault-tolerant Kafka architecture that prevents message loss during peak ingestion.
Key Takeaways
Deep understanding of distributed systems and message brokers.
Mastered Kafka internals and scalable storage architectures.
Gained experience working in highly secure, air-gapped government environments.