Virtual chunks that just work — but securely
Delivering enterprise-grade access control for virtual chunks - surprisingly subtle!
Tom Nicholas
Software Engineer
Sebastian Galkin
Staff Engineer
Sebastian Galkin
Staff Engineer
9 posts
Ship fast, break nothing: Engineering rigor in Icechunk with property and stateful testing
How we engineer rigor into Icechunk and its upstream brethren using property and stateful testing techniques. Also, a cautionary tale.
Deepak Cherian
Forward Deployed Engineer
Sebastian Galkin
Staff Engineer
Filtered Subscriptions: Fine-Grained Cloud-Native Data Sharing
Earthmover's filtered subscriptions allow data providers to create secure, read-only views into multidimensional data cubes, enabling more granular cloud-native data exchange between provider and consumer.
Ryan Abernathey
CEO & Co-founder
Sebastian Galkin
Staff Engineer
Lindsey Nield
Software Engineer
Announcing Icechunk 2: Better Consistency, Performance, and Reliability for Tensor Storage
When we released Icechunk 1.0 last July, we declared it production-ready and committed to format stability. Since then, adoption has exceeded our expectations. Teams across weather forecasting, climate science, neuroscience, and AI/ML have pushed Icechunk into scenarios we didn't fully anticipate--r
Ryan Abernathey
CEO & Co-founder
Sebastian Galkin
Staff Engineer
Evolving our Tensor Storage Engine: A Preview of Icechunk 2
A preview of Icechunk 2, featuring node rename, chunk reindexing, rectilinear grids, repository-level metadata, and significant performance improvements with a smooth migration path from Icechunk 1.
Sebastian Galkin
Staff Engineer
Everything you need to know about Icechunk garbage collection
A practical guide to Icechunk's garbage collection and expiration operations, explaining when and how to safely reclaim storage from unused snapshots and dangling objects.
Sebastian Galkin
Staff Engineer
Icechunk: Efficient storage of versioned array data
Icechunk stores versioned array data efficiently by never copying or rewriting existing chunks, so each new version only consumes storage for the data that actually changed. Older versions can be expired and garbage-collected when they are no longer needed.
Sebastian Galkin
Staff Engineer
Learning about Icechunk consistency with a clichéd but instructive example
A practical walkthrough of how Icechunk uses transactions and conflict detection to guarantee data consistency when multiple processes write concurrently. The post demonstrates optimistic concurrency control and the rebase workflow using a bank-account transfer example.
Sebastian Galkin
Staff Engineer
Exploring Icechunk scalability: untangling S3's prefix story
Demystifying how S3 prefix sharding actually works and demonstrating that Icechunk can scale to hundreds of thousands of requests per second, far beyond the single-prefix limit.
Sebastian Galkin
Staff Engineer
Related Posts
-
Multiscale overviews in Arraylake: GeoZarr-Powered Pyramids in Flux - and How to Generate Them in Icechunk Jun 16, 2026 -
The University of Southern Mississippi makes 90 million ocean profiles conversational: AQUAVIEW's approach to the World Ocean Database Jun 15, 2026 - Virtual chunks that just work — but securely Jun 10, 2026
-
Icechunk Adopted by the National Weather Service: Earthmover Joins Booz Allen on NWS CIRRUS Jun 4, 2026
Blog Tags
access controls ai AI weather AIFS announcement arraylake Arraylake Big Data blog post blog-post case study case-study chat cirrus climate risk cloud cloud native conference data cube data infrastructure data workflows dev note earth observation earth-observation energy engineering enterprise era5 event FAIR feature release flux fundamentals garbage collection geospatial GOES-R hypothesis icechunk Icechunk IFS LangChain LangGraph marketplace mcp NASA natural language processing NCEI netCDF NetCDF noaa NOAA nws oceanography open data open science open source performance property-testing python Python Radar rust Rust security serverless shuttle tabular data technical tensorops tensors testing third-wheel toxiproxy vector data virtual-zarr virtualizarr weather wildfire WOD xarray Xarray zarr Zarr