ISRO - Streaming Large Satellite Images to Memory
POC for streaming massive satellite images directly into memory - bypassing disk I/O for faster processing and improved data security.
Project Snapshot
Role
Data Engineer Research Intern
Organization
ISRO - National Remote Sensing Centre (NRSC), Hyderabad
Timeline
February – June 2025
Status
Research / POC
Deployment
POC / Research
Tech Stack
Impact & Results
- Eliminated disk I/O bottleneck for large satellite image processing
- Improved data security in air-gapped environments
A proof-of-concept that streams massive satellite images directly into memory. This bypasses traditional disk I/O bottlenecks, enabling faster processing and improved data security in air-gapped environments.
The Problem
Satellite images at ISRO can be tens of gigabytes each. Writing them to disk before processing creates severe I/O bottlenecks and increases security exposure in secure, air-gapped environments.
The Solution
A memory-mapped streaming pipeline that loads massive satellite image data directly into RAM for immediate processing, skipping intermediate disk writes.
Architecture & Implementation
Data Streaming: Implemented a memory-mapped streaming pipeline for high-speed data access.
Processing: Loaded image chunks directly into RAM for immediate geospatial and visual processing.
Data Used: Massive (tens of GBs) satellite image files.
Tech Stack: Python, NumPy, Rasterio, HDF5.
Challenges & Solutions
Memory Limits: Managing memory limits when mapping massive (tens of GB) arrays.
Chunking: Designing efficient chunking and caching strategies for memory-mapped I/O.
Integration: Ensuring the streaming pipeline integrated smoothly with existing processing scripts.
Key Takeaways
Low-level Python memory management and optimization.
HDF5 and memory-mapped file techniques.
Identifying and resolving I/O bound bottlenecks in geospatial processing.