The challenge
Handling massive increases in data volume
We have been acquiring and archiving satellite data for the Australian Government since 1979, underpinning a wide variety of research programs ranging from inland water, bushfire and land use mapping, ocean colour monitoring and minerals exploration.
The traditional approach for storing satellite Earth observation data is to store small scenes as individual files that are manageable on desktop computers or workstations, resulting in millions of gigabytes or petabytes of data being acquired.
This approach makes the data difficult to work with on lengthy projects or large regional areas such as the Murray-Darling Basin, often resulting in significant and costly delays to research.
Further, with new satellites expected to come online from Europe, Japan and the United States over the next decade, the volume of data is expected to increase exponentially, producing multi-petabyte datasets.
Our response
Improving our data facilites
Our researchers, in collaboration with Geoscience Australia and the National Computational Infrastructure (NCI), have further developed a facility (originally created by Geoscience Australia) to house and process petabytes of satellite Earth observation data from multiple sensors.
The "Open Data Cube", a high-performance data management system for multi-platform satellite data, provides a more efficient and flexible programming interface which will significantly simplify data access.
Users are able to provide their own algorithms into the analysis framework, and for many applications it allows parallel computation across thousands of processes and a petabyte of data, enabling scientists to use the Earth observation data more efficiently, measuring changes or mapping new aspects of the Australian environment and surrounding oceans.
The results
Putting the Data Cube to use
For the initial stage of its further development, the Data Cube was used by a small number of our researchers to test and verify the quality of the source data, ease of use and to ensure that the infrastructure itself could handle the load.
Following this process, the wider Australian earth observation community was provided with general access in July 2014.
This project was co-funded by the CSIRO's earth observation science area, in collaboration with Geoscience Australia and the NCI.