Announcing the Earthmover Data Marketplace

At Earthmover, we’re on a mission to empower people to use scientific data to solve humanity’s greatest challenges. With today’s launch of the Earthmover Data Marketplace, we’re taking a huge step towards that goal. We’re launching the marketplace with a fantastic group of data providers, including Brightband, Dynamical.org, Zeus AI, Beyond Weather, Planette, Climate Uncertainty Lab, and the Subseasonal Consortium.

The Problem

AI is transforming how environmental data are both produced and consumed. Today, modern data teams want their weather and climate data in analysis-ready, cloud optimized (ARCO) data cubes for easy integration into high-performance analytics and AI workflows. But the medium of exchange for weather and climate data over the past 30 years has been the same: mountains and mountains of GRIB and NetCDF files. This places a huge burden on data engineers to ingest, transform, and optimize their datasets for analytics and AI workloads.

At Earthmover, we’ve watched over and over as our customers, and the scientific community broadly, have struggled to ingest hundreds of terabytes of weather forecasts. And what’s crazy is that most of them are ingesting the same exact datasets from the same providers!

Our Solution

We knew we needed to do something about this problem in order to unlock the real potential of cloud computing and help advance critical research and applications. We looked around and saw that many teams–including many of our customers–were already doing the work to generate production grade ARCO weather and climate datasets. To support, expand, and incentivize this critical activity, we created the Earthmover Data Marketplace–the world’s first marketplace for AI-ready weather and climate data.

Earthmover Data Marketplace offers a modern, cloud-native data sharing experience for high-velocity gridded weather, climate, and Earth-observation datasets.  

With the Data Marketplace, data users can get instant access to clean, analysis-ready data from trusted providers. This includes key reference forecasts from ECMWF and NOAA, as well as some exciting new AI-based forecasts from our amazing data partners—all provided in the high-performance cloud-native Icechunk format and ready for immediate integration into computationally demanding workflows.

No downloads, no ingestion. All ready to use immediately.

Open-Source Icechunk at the Core

Earthmover Data Marketplace is different from existing data portals because it’s built around analysis-ready data cubes, rather than individual files, using the open-source Icechunk format. Our unique “subscription” model allows users to receive incremental updates to data cubes with low latency, making it ideal for products like weather forecasts which publish updates on an hourly basis. This is technically possible thanks to Icechunk’s ACID transactions and snapshot history. And thanks to Icechunk’s best in class I/O performance, the data are ready to support high-throughput processing and training pipelines.

From the data provider perspective, our decentralized, open-source, cloud-agnostic architecture allows the provider to retain full control and sovereignty over their data while also providing ease of access for end users. Providers can choose to supply free, open data or set custom pricing, licensing, and SLAs for proprietary datasets.

The Data Providers

We’re excited to use the marketplace to support both commercial applications and research. In fact, they go hand in hand, as you can see from our launch partners. We’re proud to be working with some of the top weather startups and academic research groups to launch the marketplace with some amazing datasets. 

• Brightband is providing several ECMWF forecast datasets–including a low-latency IFS HRES product suitable for production use in energy and commodities trading–as well as a four-year archive of Machine-Learning Weather Prediction (MWLP) forecast.
• Dynamical.org is contributing their free and openly licensed operational ARCO weather data cubes, including NOAA GEFS, GFS, and HRRR forecast and analysis, plus an ECMWF IFS Ensemble forecast.
• Zeus AI is providing their AI-powered Helios Solar Irradiance forecast in both archival and near-real-time configuration
• Beyond Weather has produced a fine-tuned version of the ECMWF AIFS model optimized for 100m wind predictions.
• Planette is providing their flagship seasonal forecast based on a hybrid physics / AI model.
• Climate Uncertainty Lab is providing their brand new bias-corrected downscaled climate projection massive ensemble, a major leap forward in understanding climate risk uncertainty.
• The Subseasonal Consortium is contributing a multi-model collection of sub-seasonal forecasts from a range of research labs.

We’re thrilled to have these amazing partners joining us for the launch. And this is just the beginning. We’re also continuing to onboard new providers on a regular basis, so reach out if you’re interested in getting involved.

The Impact

Data marketplace is a gamechanger for teams in energy, commodities, agriculture, insurance, and climate risk. You can now obtain instant access to the AI-ready weather and climate data you need, bypassing months of custom engineering and endless ongoing maintenance. Our goal is to massively accelerate your development roadmap.

Marketplace is just one piece of the Earthmover Platform. Together with Arraylake, our cloud-native array data management and governance layer, and Flux, our geospatial API query layer, the platform offers everything teams need to build and maintain mission-critical scientific data systems for research and operations.

Marketplace and Earthmover are Open to the World

Until today, our platform has been available only to Earthmover paying customers. With the launch of Marketplace, we’re also opening up the platform to anyone in the world! Anyone can sign up for an account and access the free and open datasets in the Marketplace.

So head over the https://app.earthmover.io/marketplace and let us know what you think!