Icechunk-ERA5: a daily updating, performance-optimized ARCO data cube, with 86 years of 43 surface and pressure-level variables. Available now on the Earthmover Data Marketplace .
Tag

#Zarr

21 posts

Variable length chunks in Zarr
Blog Post

Variable length chunks in Zarr

A new extension to Zarr just landed: the rectilinear chunk grid lets you specify arbitrarily sized chunks along each axis, aligning chunk boundaries with the natural structure of your data instead of forcing a regular grid.

Joe Hamman
Joe Hamman

CTO & Co-founder

Max Jones
Max Jones

Cloud Engineer @ DevSeed

Davis Bennet
Davis Bennet

Software Engineer (Freelance)

How Kettle Uses Earthmover to Power Wildfire Risk Modeling at Scale
Case Study

How Kettle Uses Earthmover to Power Wildfire Risk Modeling at Scale

The Company Kettle is not a typical insurance company. Using AI to build smarter insurance products, Kettle provides insurance for property owners in areas affected by catastrophic climate events, with a particular focus on wildfire. Their AI models consume over 130 terabytes of satellite, weather,

Margaret Francis
Margaret Francis

COO

I/O-Maxing Tensors in the Cloud
Blog Post

I/O-Maxing Tensors in the Cloud

Zarr Python with Icechunk or Obstore now fully saturates the network between EC2 and S3, achieving the physically maximum possible throughput for reading and writing tensor data in the cloud. Benchmarks compare Zarr, Tensorstore, TileDB, and Parquet stacks across a range of chunk sizes and instance types.

Ryan Abernathey
Ryan Abernathey

CEO & Co-founder

Building the Future of Scientific Data at the Zarr Summit
Blog Post

Building the Future of Scientific Data at the Zarr Summit

Earthmover co-organizes the Zarr Summit in Rome, bringing together developers and adopters to advance the open-source cloud-native array format as adoption accelerates across major organizations like ESA, NASA, and NVIDIA.

Ryan Abernathey
Ryan Abernathey

CEO & Co-founder

Multi-Player Mode: Why Teams That Use Zarr Need Icechunk
Blog Post

Multi-Player Mode: Why Teams That Use Zarr Need Icechunk

Zarr lacks built-in support for concurrent readers and writers, leading to inconsistent reads and conflicting writes in team settings. Icechunk solves this by adding atomic updates, consistent snapshots, and Git-like version control on top of Zarr.

Lindsey Nield
Lindsey Nield

Software Engineer

Icechunk 1.0: Production-Grade Cloud-Native Array Storage Is Here
Announcement

Icechunk 1.0: Production-Grade Cloud-Native Array Storage Is Here

Icechunk 1.0 is now stable and production-ready, bringing transactional safety, efficient versioning, high-performance Rust-based I/O, and virtual references for HDF5 and NetCDF to cloud-native array storage. The release includes manifest splitting, distributed writes, conflict resolution, and a 30 TB ERA5 sample dataset.

Ryan Abernathey
Ryan Abernathey

CEO & Co-founder

Zarr takes Cloud-Native Geospatial by storm
Blog Post

Zarr takes Cloud-Native Geospatial by storm

At the 2025 Cloud-Native Geospatial conference, Zarr adoption was surging across the geospatial domain, with Copernicus Sentinel, USGS Landsat, Google Earth Engine, and ESRI ArcGIS all embracing the format for cloud-optimized array data.

Joe Hamman
Joe Hamman

CTO & Co-founder

Accelerating Xarray with Zarr-Python 3
Blog Post

Accelerating Xarray with Zarr-Python 3

zarr-python’s performance paradox Last month, we released Zarr-Python 3.0 - a ground-up rewrite of the library (read more about it in this post). Beyond the exciting new features in Zarr V3, we put a lot of work into addressing some long standing performance issues with Zarr-Python 2. With the improvements described in this blog post, we’ve achieved a 14x speedup in loading the ARCO ERA5 dataset! Zarr-Python 2 had a paradoxical performance quirk; although the library could generate massive petabyte-scale datasets, it struggled to perform well when managing large or highly nested hierarchies. For example, listing the contents of a large Zarr group could be painfully slow, particularly if that Zarr group was stored on a high latency storage backend. Zarr users would experience this as long

Davis Bennet
Davis Bennet

Software Engineer (Freelance)

Zarr-Python 3 is here!
Announcement

Zarr-Python 3 is here!

Zarr-Python 3.0 is released with full support for the Zarr V3 specification, chunk-sharding for more flexible storage, major performance improvements from a fully asynchronous core, and a modernized extensible codebase.

Joe Hamman
Joe Hamman

CTO & Co-founder

Announcing Icechunk!
Announcement

Announcing Icechunk!

Earthmover announces Icechunk, an open-source transactional storage engine for Zarr that brings ACID transactions, time travel, data versioning, and high-performance Rust-based I/O to multidimensional array data in cloud object storage.

Ryan Abernathey
Ryan Abernathey

CEO & Co-founder

Toward Zarr-Python 3.0
Blog Post

Toward Zarr-Python 3.0

The Zarr-Python project is undergoing a major refactor toward version 3.0, bringing full support for the Zarr V3 specification, new asynchronous APIs for better performance, and a modernized plugin system for codecs and storage backends.

Joe Hamman
Joe Hamman

CTO & Co-founder

← All posts