transactional storage for tensors

"Structured data" usually means tabular data — rows, columns, the relational model. But what makes data structured is really that it conforms to a data model that eliminates redundancy. The relational model is one such model. For data about the physical world — weather, climate, satellite imagery, microscopy, genomics — the right model is the multidimensional array, or tensor.

That model has a name: the Unidata Common Data Model, which underlies NetCDF and HDF5, and there are exabytes of data in the wild that use it. A dataset is a set of variables, each an array with a homogeneous type and named dimensions, indexed by small coordinate variables. The coordinates are stored parsimoniously — one list per dimension, not one value per cell.

Compression can hide some of that duplication, but it can't remove it from the data model — and in a table, locating a single point in space and time means scanning every row. Keep the dimensions and you keep efficient indexing; flatten them and you pay for it on every query.

The formats predate the cloud

NetCDF, HDF5, GRIB, and TIFF were designed for filesystems, where seeking is nearly free. Object storage is different: there is no "open a file", only requests over the network, each carrying tens of milliseconds of latency. Because these formats scatter their metadata throughout the file, simply discovering what's inside one means a long sequence of dependent requests. In one benchmark, reading just the metadata of a single 1 GB NetCDF file on S3 took 502 requests to move 267 KB — an effective 7 KB/s, slower than dial-up. And that's one file; real datasets have thousands.

Zarr: Parquet for arrays

Zarr keeps the array data model and fixes the layout for object storage. The metadata lives in its own known location, so one request tells you everything about the dataset's structure. The arrays are split into chunks, each an independently addressable object, so readers fetch exactly the chunks a query touches — thousands of them in parallel, saturating the enormous throughput object storage offers. Groups nest like directories, and every level carries its own metadata, so datasets stay self-describing and lazily navigable.

Zarr gives you the format. What it doesn't give you is a database — and that's where Icechunk comes in.

Zarr is a cloud-optimized format for multidimensional arrays that is a significant advance over legacy formats like NetCDF, GRIB, and TIFF. However, Zarr has three critical limitations:

• Safety — You can accidentally delete or corrupt your data in an unrecoverable way.

• Consistency — Concurrent readers and writers can see partial, inconsistent data.

• Reproducibility — Data modifications happen silently with no version tracking.

These limitations mean Zarr cannot be reliably used as a database — a critical capability for modern data-intensive workflows like weather forecasting, climate modeling, and geospatial analysis, where multiple teams need to safely read and update shared datasets.

Icechunk provides git-like snapshots combined with ACID transactions. Every write operation creates an immutable snapshot, while branches and tags provide familiar version-control semantics:

• Safety — Recover from corrupted or accidentally deleted data using version history.

• Consistency — Readers always see complete, valid snapshots — never partial writes.

• Reproducibility — Reference any version of your data permanently via commits or tags.

# Write and commit atomically - readers never see partial updates
with repo.transaction("main", message="Add new forecast") as store:
    group = zarr.open_group(store)
    group["temperature"][:] = new_temp_data
    group["pressure"][:] = new_pressure_data
    # Both arrays updated together or not at all

# Time travel to any previous version
historical_session = repo.readonly_session(snapshot_id="abc123")

# Reproducible analysis with permanent references
repo.create_tag("v1.0-release", snapshot_id="abc123")  # Immutable reference
session = repo.readonly_session(tag="v1.0-release")