Blog icon

The challenge

Handling massive increases in data volume

We have been acquiring and archiving satellite data for the Australian Government since 1979, underpinning a wide variety of research programs ranging from inland water, bushfire and land use mapping, ocean colour monitoring and minerals exploration.

Satellite data is integral to mapping the Australian environment and surrounding oceans.

The traditional approach for storing satellite Earth observation data is to store small scenes as individual files that are manageable on desktop computers or workstations, resulting in millions of gigabytes or petabytes of data being acquired.

This approach makes the data difficult to work with on lengthy projects or large regional areas such as the Murray-Darling Basin, often resulting in significant and costly delays to research.

Further, with new satellites expected to come online from Europe, Japan and the United States over the next decade, the volume of data is expected to increase exponentially, producing multi-petabyte datasets.

Our response

Improving our data facilites

Our researchers, in collaboration with Geoscience Australia and the National Computational Infrastructure (NCI), have further developed a facility (originally created by Geoscience Australia) to house and process petabytes of satellite Earth observation data from multiple sensors.

The "Open Data Cube", a high-performance data management system for multi-platform satellite data, provides a more efficient and flexible programming interface which will significantly simplify data access.

Users are able to provide their own algorithms into the analysis framework, and for many applications it allows parallel computation across thousands of processes and a petabyte of data, enabling scientists to use the Earth observation data more efficiently, measuring changes or mapping new aspects of the Australian environment and surrounding oceans.

The results

Putting the Data Cube to use

For the initial stage of its further development, the Data Cube was used by a small number of our researchers to test and verify the quality of the source data, ease of use and to ensure that the infrastructure itself could handle the load.

Following this process, the wider Australian earth observation community was provided with general access in July 2014.

This project was co-funded by the CSIRO's earth observation science area, in collaboration with Geoscience Australia and the NCI.

Do business with us to help your organisation thrive

We partner with small and large companies, government and industry in Australia and around the world.

Contact us now to start doing business

Contact us

Find out how we can help you and your business. Get in touch using the form below and our experts will get in contact soon!

CSIRO will handle your personal information in accordance with the Privacy Act 1988 (Cth) and our Privacy Policy.

First name must be filled in

Surname must be filled in

I am representing *

Please choose an option

Please provide a subject for the enquriy

0 / 100

We'll need to know what you want to contact us about so we can give you an answer

0 / 1900

You shouldn't be able to see this field. Please try again and leave the field blank.