Your data layer matters more than ever
Agents can build the app. Turning raw files into a dataset still takes a human, so share the dataset, and do it without copying the data.
Ian Hunt-Isaak
Xarray Community Developer
blog · tag
#Icechunk
42 posts
Fundamentals: What is zero-copy ingest?
You want to use Zarr, but your data is in HDF5, TIFF, or ND2. Zero-copy ingest lets you read it as Zarr without converting it.
Ian Hunt-Isaak
Xarray Community Developer
Convert Zarr to Icechunk with just a few clicks!
Arraylake makes it trivial to ingest and view existing Zarr stores in the cloud.
Tom Nicholas
Software Engineer
How to make scientific data archives AI-ready
Unlocking scientific data archives for human, agentic and AI model usage requires a truly scalable data storage architecture. We show from first principles what architecture is necessary.
Tom Nicholas
Software Engineer
A Data Layer for Scientific AI: Icechunk for Agentic Single-Cell Biology Workflows
Agents work best when they can branch, explore in parallel, and converge. Using examples from single-cell biology, we show how Icechunk turns a shared cloud Zarr store into a versioned, transactional system of record for agentic science.
Ryan Abernathey
CEO & Co-founder
Luiz Irber
Software Engineer
Ian Hunt-Isaak
Xarray Community Developer
NVIDIA: Building a Low-Latency Earth-2 Nowcasting Pipeline with Earthmover
NVIDIA and Earthmover built a low-latency nowcasting workflow that turns Meteosat satellite observations into a two-hour nowcast using Earth-2 StormScope-EU, and serves it directly from versioned data through map tiles and SQL.
Negin Sobhani
Senior Machine Learning Performance Engineer, NVIDIA
Aaron Spring
Data and AI Freelancer
Jussi Leinonen
Senior Solution Architect, NVIDIA
Marius Koch
Senior Solution Architect, NVIDIA
Weather Forecast Data: Icechunk vs. Iceberg Head-to-Head
TLDR: Operational weather forecast data workloads built on Icechunk are 10-100x faster and 5-8x cheaper than Iceberg. Warning: This article is long. It's written for a system architect planning to expend significant budget on weather data infrastructure and looking to deeply understand the options a
Ryan Abernathey
CEO & Co-founder
Science data comes in tables and tensors
Apache Iceberg comes to Arraylake. Tables now live alongside Icechunk arrays in the same catalog, under the same permissions, in your own buckets — queryable from DuckDB, Spark, Trino, Snowflake, and Zax-SQL.
Joe Hamman
CTO & Co-founder
Matt Iannucci
Staff Engineer
Our compute roadmap: introducing Zax SQL
We're introducing Zax, a new query and compute engine for multidimensional scientific data, and Zax-SQL, the first interface to it. Zax plans your computation for you, so you are no longer responsible for tuning chunk sizes, worker counts, and memory limits.
Ryan Abernathey
CEO & Co-founder
Brian Davis
Head of Product
Joe Hamman
CTO & Co-founder
Deepak Cherian
Principal Research Engineer
Sebastian Galkin
Head of Engineering
Making open-source AI weather forecasting models easy to run
A joint post from Earthmover and Hugging Face on running open-source AI weather forecasting models like Aurora using free ERA5 data from the Earthmover Marketplace — with a live demo space and a full local tutorial.
Emma Scharfmann
ML for Science, Hugging Face
Aaron Spring
ML, AI and geospatial data engineering
Ryan Abernathey
CEO & Co-founder
Kashif Rasul
AI Research Science, Hugging Face
Having it all: virtual and materialized data products
A guest post from dynamical.org on how virtual Icechunk Zarrs complement materialized products to deliver low-latency, complete, and fast-access weather data — plus the low-latency NOAA HRRR dataset it describes, now on the Earthmover Data Marketplace.
Alden Keefe Sampson
dynamical.org
Marshall Moutenot
dynamical.org
Virtually Gribberish - Bringing Icechunk clarity to GRIB archives
GRIB data is notoriously hard to work with, so we made it easy to create virtual Icechunk stores from GRIB archives such as the NOAA National Blend of Models, allowing cloud-optimized access to GRIB data archives without copying the data.
Tom Nicholas
Software Engineer
Matt Iannucci
Staff Engineer
The University of Southern Mississippi makes 90 million ocean profiles conversational: AQUAVIEW's approach to the World Ocean Database
How AQUAVIEW reorganized NOAA's World Ocean Database around instrument types, powered it with Arraylake, and built a natural language interface that turns hours-long data workflows into seconds.
Henry Jones PhD
Director of Research Development and Scientific Entrepreneurship, University of Southern Mississippi
Joshua Hill PhD
Director, Institute for Advanced Analytics and Society, University of Southern Mississippi
Virtual chunks that just work — but securely
Delivering enterprise-grade access control for virtual chunks - surprisingly subtle!
Tom Nicholas
Software Engineer
Sebastian Galkin
Head of Engineering
Icechunk Adopted by the National Weather Service: Earthmover Joins Booz Allen on NWS CIRRUS
Earthmover has teamed up with Booz Allen Hamilton on NWS CIRRUS — NOAA's new cloud-based operational data platform for the National Weather Service. Icechunk will serve as a core data format for the operational data lake.
Joe Hamman
CTO & Co-founder
Old format, no problem!: Cloud-optimizing the GOES-16 archive as Virtual Zarr
Access billions of chunks of satellite imagery as a single Zarr store, without copying any data!
Tom Nicholas
Software Engineer
Low Latency Icechunk ERA5 Now Available on the Earthmover Data Marketplace
We released Icechunk-ERA5, a performance-optimized, daily updating ERA5 data cube available now in the Earthmover data marketplace.
Ryan Abernathey
CEO & Co-founder
Ship fast, break nothing: Engineering rigor in Icechunk with cross-version stateful compatibility testing
How we used third-wheel and Hypothesis to interleave Icechunk 1.x and 2.x in a single stateful test — and the bugs that fell out.
Ian Hunt-Isaak
Xarray Community Developer
Beyond Weather: On-demand visualizations of fine tuned AIFS models over NetCDF workarounds in one week
The Company: Long Range AI Weather Prediction Beyond Weather is an Amsterdam-based AI weather forecasting startup with its roots in academic research. Founded in 2023 by a master's student, a PhD researcher, and a professor from the Vrije Universiteit Amsterdam's climate extremes and societal risk r
Margaret Francis
COO
Ship fast, break nothing: Engineering rigor in Icechunk with concurrency & fault injection testing
We describe our approach to rigor in Icechunk 2 including concurrency permutation testing and more!
Deepak Cherian
Principal Research Engineer
Ship fast, break nothing: Engineering rigor in Icechunk with property and stateful testing
How we engineer rigor into Icechunk and its upstream brethren using property and stateful testing techniques. Also, a cautionary tale.
Deepak Cherian
Principal Research Engineer
Sebastian Galkin
Head of Engineering
How Kettle Uses Earthmover to Power Wildfire Risk Modeling at Scale
The Company Kettle is not a typical insurance company. Using AI to build smarter insurance products, Kettle provides insurance for property owners in areas affected by catastrophic climate events, with a particular focus on wildfire. Their AI models consume over 130 terabytes of satellite, weather,
Margaret Francis
COO
Filtered Subscriptions: Fine-Grained Cloud-Native Data Sharing
Earthmover's filtered subscriptions allow data providers to create secure, read-only views into multidimensional data cubes, enabling more granular cloud-native data exchange between provider and consumer.
Ryan Abernathey
CEO & Co-founder
Sebastian Galkin
Head of Engineering
Lindsey Nield
Senior Engineer
From GeoTIFF Chaos to Cloud-Native Climate Risk: How Eoliann Built Airis on Earthmover
Eoliann builds proprietary climate risk models that estimate the physical impact of extreme weather events — floods, wildfires, and storms — on critical physical infrastructure: electricity transmission lines, substations, and gas pipelines…
Margaret Francis
COO
Announcing Icechunk 2: Better Consistency, Performance, and Reliability for Tensor Storage
When we released Icechunk 1.0 last July, we declared it production-ready and committed to format stability. Since then, adoption has exceeded our expectations. Teams across weather forecasting, climate science, neuroscience, and AI/ML have pushed Icechunk into scenarios we didn't fully anticipate--r
Ryan Abernathey
CEO & Co-founder
Sebastian Galkin
Head of Engineering
Announcing IceChunkCoin: The World's First Multi-Dimensional, Chunk-Native Blockchain Asset
Today we’re thrilled to announce IceChunkCoin, the world’s first hyper-dimensional, chunk-native blockchain asset.
Ryan Abernathey
CEO & Co-founder
Cursed venvs, Confident Releases: Testing Icechunk Across Major Versions
Earthmover built third-wheel, an open-source tool that rewrites Python wheels to install multiple versions of a library in one environment, enabling cross-version compatibility testing for the Icechunk V2 release.
Ian Hunt-Isaak
Xarray Community Developer
Evolving our Tensor Storage Engine: A Preview of Icechunk 2
A preview of Icechunk 2, featuring node rename, chunk reindexing, rectilinear grids, repository-level metadata, and significant performance improvements with a smooth migration path from Icechunk 1.
Sebastian Galkin
Head of Engineering
I/O-Maxing Tensors in the Cloud
Zarr Python with Icechunk or Obstore now fully saturates the network between EC2 and S3, achieving the physically maximum possible throughput for reading and writing tensor data in the cloud. Benchmarks compare Zarr, Tensorstore, TileDB, and Parquet stacks across a range of chunk sizes and instance types.
Ryan Abernathey
CEO & Co-founder
From 10 Minutes to 10 Seconds: How Woods Hole Scientists used Icechunk to Optimize Ocean Data Access
Woods Hole scientists reduced ocean profile data access from 10 minutes to 10 seconds by converting their OPeNDAP-served NetCDF files to Icechunk repositories on AWS S3.
Iury Simoes-Sousa
Postdoctoral Investigator, WHOI
Multi-Player Mode: Why Teams That Use Zarr Need Icechunk
Zarr lacks built-in support for concurrent readers and writers, leading to inconsistent reads and conflicting writes in team settings. Icechunk solves this by adding atomic updates, consistent snapshots, and Git-like version control on top of Zarr.
Lindsey Nield
Senior Engineer
Radar DataTree: Transforming thousands of scans into a single cohesive model
Introducing the Radar DataTree, a new data model that organizes thousands of fragmented weather radar scans into a single time-aware, cloud-native, version-controlled dataset using xarray-datatree, Zarr, and Icechunk.
Alfonso Ladino-Rincon
Data Scientist
Icechunk 1.0: Production-Grade Cloud-Native Array Storage Is Here
Icechunk 1.0 is now stable and production-ready, bringing transactional safety, efficient versioning, high-performance Rust-based I/O, and virtual references for HDF5 and NetCDF to cloud-native array storage. The release includes manifest splitting, distributed writes, conflict resolution, and a 30 TB ERA5 sample dataset.
Ryan Abernathey
CEO & Co-founder
Everything you need to know about Icechunk garbage collection
A practical guide to Icechunk's garbage collection and expiration operations, explaining when and how to safely reclaim storage from unused snapshots and dangling objects.
Sebastian Galkin
Head of Engineering
Fundamentals: What Is Zarr? A Cloud-Native Format for Tensor Data
Zarr is an open-source, cloud-native protocol for storing chunked, compressed N-dimensional arrays. This guide covers how Zarr works, its ecosystem of tools like Xarray and Icechunk, and when to use it for large-scale scientific and ML data.
Lindsey Nield
Senior Engineer
Icechunk: Efficient storage of versioned array data
Icechunk stores versioned array data efficiently by never copying or rewriting existing chunks, so each new version only consumes storage for the data that actually changed. Older versions can be expired and garbage-collected when they are no longer needed.
Sebastian Galkin
Head of Engineering
Zarr takes Cloud-Native Geospatial by storm
At the 2025 Cloud-Native Geospatial conference, Zarr adoption was surging across the geospatial domain, with Copernicus Sentinel, USGS Landsat, Google Earth Engine, and ESRI ArcGIS all embracing the format for cloud-optimized array data.
Joe Hamman
CTO & Co-founder
Learning about Icechunk consistency with a clichéd but instructive example
A practical walkthrough of how Icechunk uses transactions and conflict detection to guarantee data consistency when multiple processes write concurrently. The post demonstrates optimistic concurrency control and the rebase workflow using a bank-account transfer example.
Sebastian Galkin
Head of Engineering
Fundamentals: What is Cloud-Optimized Scientific Data?
Why traditional scientific file formats like NetCDF perform poorly on cloud object storage, and how cloud-optimized formats like Zarr and Icechunk solve the problem by separating metadata and chunking data.
Tom Nicholas
Software Engineer
Exploring Icechunk scalability: untangling S3's prefix story
Demystifying how S3 prefix sharding actually works and demonstrating that Icechunk can scale to hundreds of thousands of requests per second, far beyond the single-prefix limit.
Sebastian Galkin
Head of Engineering
Solving NASA's Cloud Data Dilemma: How Icechunk Revolutionizes Earth Data Access
Earthmover and Development Seed partnered with NASA to pilot Icechunk, an open-source tensor storage engine that enables 100x faster cloud-native data access for archival Earth science datasets without costly data migration.
Ryan Abernathey
CEO & Co-founder
Announcing Icechunk!
Earthmover announces Icechunk, an open-source transactional storage engine for Zarr that brings ACID transactions, time travel, data versioning, and high-performance Rust-based I/O to multidimensional array data in cloud object storage.
Ryan Abernathey
CEO & Co-founder
Related Posts
Blog Tags
access controls access-control agents ai AI weather AIFS announcement architecture arraylake Arraylake aurora Big Data biology blog post blog-post case study case-study chat cirrus climate risk cloud cloud native compliance compute conference data archives data cube data infrastructure data workflows dev note dynamical earth observation earth-2 earth-observation energy engineering enterprise era5 event FAIR feature release financial-services flux forecast fundamentals garbage collection geospatial GOES-R governance grib guest-post hrrr hugging-face hypothesis iceberg icechunk Icechunk IFS LangChain LangGraph marketplace mcp NASA natural language processing NCEI netCDF NetCDF noaa NOAA nowcasting nvidia nws oceanography open data open science open source performance product property-testing public-sector python Python Radar rust Rust scverse security serverless shuttle single-cell soc2 sql tables tabular data teams technical tensorops tensors testing third-wheel toxiproxy trust vector data virtual-zarr virtualizarr virtualization weather wildfire WOD xarray Xarray zarr Zarr zax zero-copy-ingest