We have been acquiring and archiving satellite data for the Australian Government since 1979, underpinning a wide variety of research programs ranging from inland water, bushfire and land use mapping to ocean colour monitoring and minerals exploration.
The traditional approach for storing satellite Earth observation data is to store small scenes as individual files that are manageable on desktop computers or workstations, resulting in millions of gigabytes or petabytes of data being acquired.
This approach makes the data difficult to work with on lengthy projects or large regional areas such as the Murray-Darling Basin, often resulting in significant and costly delays to research.
With new satellites expected to come online from Europe, Japan and the United States over the next decade, the volume of data is expected to increase exponentially, producing multi-petabyte datasets.
Handling massive increases in data volume
Our researchers, in collaboration with Geoscience Australia and the National Computational Infrastructure (NCI), further developed the 'Australian Geoscience Data Cube' originally created by Geoscience Australia that houses and processes petabytes of satellite Earth observation data from multiple sensors.
Since 2017, several international partners (NASA, USGS, CEOS, UK's Satellite Applications Catapult) have also joined the partnership and the platform is now called the Open Data Cube. It provides an efficient and flexible programming interface that significantly simplifies access to freely available satellite data about our world.
Users are able to add their own algorithms into the analysis framework, and for many applications it allows parallel computation across thousands of processes and a petabyte of data, enabling scientists to use the Earth observation data more efficiently, measuring changes or mapping new aspects of the Australian environment and surrounding oceans.
Putting the Open Data Cube to use
We're using the Open Data Cube to develop the next generation of high-performance data analytics platform for Earth observation data. Called the Earth Analytics and Science Innovation or EASI, it 'turbo-charges' the capacity to process and integrate huge amounts of Earth observation data with other geospatial information and models.
Through EASI, we aim to:
- assist business growth in Earth observation applications through innovation and partnerships
- equip and train government, industry and research in the use of Earth observation data and analytics
- supply digital services from CSIRO science to local, regional and global end-users.
Our partner Geoscience Australia has developed Digital Earth Australia, the Australian government's implementation of the open source analysis platform developed as part of the Open Data Cube initiative.