ISRO Satellite Data Archival, Monitoring & Notification System
Secure, scalable system for archiving petabyte-scale satellite data at ISRO NRSC - with real-time monitoring, event streaming via Kafka, and automated notifications.
Project Snapshot
Role
Data Engineer Research Intern
Organization
ISRO - National Remote Sensing Centre (NRSC), Hyderabad
Timeline
February – June 2025
Status
Completed
Team Size
2 (co-developed with Arushima; mentored by ISRO scientists)
Deployment
Deployed within ISRO NRSC's secure internal infrastructure
Tech Stack
Impact & Results
- Handles terabytes of national satellite data
- Near real-time event monitoring and alerting
- Research paper co-authored (pending formal submission)
- ISRO Letter of Recommendation issued by C. Pradeep, Scientist/Engineer, NRSC
- ISRO Project Completion Certificate issued by Dr. G. Prasad, Group Head ICIG/DPA, NRSC
An automated system built during a research internship at ISRO NRSC for managing, screening, and monitoring incoming satellite imagery during live data ingestion. The system handles massive datasets and uses a Computer Vision pipeline to automatically screen regions in near real-time.
The Problem
At ISRO NRSC, incoming satellite image strips needed to be automatically screened at ingestion time to flag which regions had already been surveyed and recorded versus which were new and needed deeper analysis. This had to run continuously as part of a live data dump—as much a speed problem as an accuracy problem.
The Solution
Developed a CNN-based classifier optimized specifically for a low false-negative rate, comfortably running inside the strict 5-second constraint per input. The image strips are long, narrow parallel cuts of a continuous surface area. While it successfully processes ~1,500 high-density image strips, a known limitation is that regions falling across a strip boundary are evaluated independently.
Architecture & Implementation
Dataset: ~1,500 high-density image strips (RGB, infrared, UV), with ~1,000 candidate regions per strip ranging from 1 km to 25 km in diameter.
First Attempt: Pixel-level Gaussian filtering + histogram comparison (highly accurate but too slow at 14.68s per input).
Final Architecture: A scalable CNN-based Computer Vision pipeline optimized for a low false-negative rate to ensure genuinely new regions are never missed.
Environment: Architecturally similar to the restricted production setup, operating as a service on a Linux VM.
Challenges & Solutions
Failure Under Load: Initial approach buffered stream chunks to disk and cleared them post-processing. This passed a 2-hour test but exhausted storage after ~14 hours under a sustained 24-hour test as input outpaced clearing rate.
The Fix: Reduced chunk size and increased thread count to process and clear chunks faster than new data arrived, keeping storage bounded over continuous runs.
Constraint Management: Successfully balancing the strict < 5 seconds per input latency budget with acceptable accuracy for initial screening.
Key Takeaways
Realistic Testing: A short test can completely hide a problem that only shows up under realistic, sustained load.
Trade-offs: Sometimes lowering overall accuracy is acceptable if you specifically optimize the critical metric (like false-negative rate) for a screening stage.
Distributed Processing: Fine-tuning chunk sizes and thread counts is critical to preventing resource exhaustion in continuous streaming systems.