33 posts
As a distributed team (of weather and climate geeks), we love finding ways to connect. So when a nationwide winter storm rolled through, we thought it would be fun to set up a Slack channel that tells us who it's snowing for right now. Here's how we built that in a single afternoon — without the pai
Xarray Community Developer
Engineering
The critical role of I/O in data science and AI/ML For both analytics and AI workloads, fast I/O is the foundation of good performance. Most of these workloads involve fluxing a large amount of data from storage into RAM, and then to the CPU or GPU. In the cloud, where data reside on object storage,
CEO & Co-founder
This blog was also published on the xarray blog and the napari blog. TL;DR Making Napari and Xarray work better together will enhance data visualization for both biology and geosciences. This has been long desired by the community but has not yet been implemented. At the SciPy 2025 sprints we formed
Xarray Community Developer
Next week we are convening the Zarr community in Rome, Italy for a week of fast-paced collaboration and conversation. Given the recent acceleration on Zarr adoption across major data providers in Weather Forecasting, Earth Observation and Bioimaging, this in-person event is critical for aligning sta
CEO & Co-founder
Note: this is a guest post by Woods Whole Oceanographic Institute scientist Dr. Iury Simoes-Sousa Among the many ways the ocean affects our lives, one of the most fundamental is through its ability to transport key climate variables such as heat, nutrients, and marine organisms. This transport happe
Postdoctoral Investigator, WHOI
Earthmover is launching a new open-source library xpublish-tiles that powers our new Flux Tiles service, which allows Earthmover Platform users to view their data on a slippy map with dynamically rendered tiles at lower zoom levels than was possible previously.
Forward Deployed Engineer
Engineering
Calculating climate risk metrics from ERA5 using Arraylake, Icechunk, and Xarray. Just show me the notebook! It’s here. Intro This year at NYC climate week it’s been pretty hot and humid outside. This felt ironically appropriate while we ran a participatory workshop “Open Data in Applied Risk Analys
Software Engineer
Note: this is the blog post version of a webinar we gave last month, summarized for brevity. At Earthmover, we’re closely following the AI revolution in weather forecasting. Until recently, the only way to make an accurate global-scale weather forecast was to run an expensive, physics-based numerica
CEO & Co-founder
Bring reliability, scalability, and version control to your Zarr datasets, without giving up performance. Zarr is a powerful protocol for storing large-scale, multi-dimensional arrays. It's fast, scalable, and cloud-native, which is why it's used across a variety of domains like climate science and
Software Engineer
From structure to scale, radar needs a model that organizes complete collections as time-aware, cloud-native datasets. In our second post, we looked at how new standards and open-source tools are transforming weather radar from raw binary blobs into structured, metadata-rich datasets. FM-301—an offi
Data Scientist
Earthmover is proud to announce its sponsorship of Ocean Hack Week 2025, continuing a tradition of supporting hackweeks that have long served as vibrant spaces for the Pangeo Community to meet, share ideas, and innovate on solving real-world scientific challenges. Earthmover founders Ryan and Joe ha
CEO & Co-founder
At Earthmover, we’re interested in weather radar data for two reasons: - First, radar data are uniquely valuable for our customers thanks to their ability to characterize precipitation, atmospheric turbulence, and phenomena like tornadoes and hurricanes in real time with fine spatial and temporal re
Data Scientist
Weather radar is one of the most powerful observational tools in atmospheric science. Every few minutes, it captures reflectivity, among other variables, in a sample volume that is rotated forming a high-resolution, four-dimensional dataset (x,y,z,t) that tracks storms in real time, revealing fine-s
Data Scientist
At Earthmover, we contribute to maintaining and driving forward a range of community open-source projects including Xarray and Zarr. The following post, cross-posted from the Xarray developer blog, describes new API for seasonal aggregation in Xarray. TL;DR Two new Grouper objects - SeasonGrouper an
Forward Deployed Engineer
Open-data practice in ocean/atmosphere sciences is approximately 170 years old! While it is easy to exclaim, "weather/climate are global, of course we must share data", the actual story is anything but. That story holds valuable inspiration that we can draw from as we face a significant reduction in US climate science research.
Forward Deployed Engineer
This was originally published on the Xarray blog: https://xarray.dev/blog/xarray-biology Hi! I'm Ian, a multimodal microscopist, and the new "Xarray Community Developer." I am funded by the Chan Zuckerberg Institute to support the use of Xarray in biological and biomedical applications. I believe Xa
Xarray Community Developer
We will talk about two powerful Icechunk operations: expiration and garbage collection. They are related, so we usually refer to both under the name of garbage collection or simply GC. We will explain what each of them does, why you may want to use them, and how to do it safely and effectively. The
Staff Engineer
Why scientists, data engineers, and developers are turning to Zarr Often the biggest bottleneck in your workflow isn’t your code or your hardware, but the way your data is stored. Data formats can limit–or unlock–what you’re able to do with your data. In modern science and data-intensive computing,
Software Engineer
We recently got an interesting question in Icechunk's community Slack channel (thank you Iury Simoes-Sousa for motivating this post): I'm new to Icechunk. How is the storage managed for redundant information between different versions of a data repository? Icechunk keeps your data versioned, allowin
Staff Engineer
Okay friends, it's time to take the Data Pain Survey: - Does it take your team three weeks or more to repurpose an existing dataset for a new data product? - Do you have team-A and team-B versions of the same dataset? - Do you miss delivery dates because you cannot estimate the work necessary to ser
Software Engineer
Our takeaways from the Cloud-Native Geospatial conference on Zarr's surging adoption and its impact on the future of Earth Observation data. Our team just returned from an action-packed week at the Cloud-Native Geospatial conference in beautiful Snowbird, Utah, and the key takeaway was unmistakable:
CTO & Co-founder
In this post we'll show what can happen when more than one process write to the same Icechunk repository concurrently, and how Icechunk uses transactions and conflict resolution to guarantee consistency. For this, we'll use a commonplace example: bank account transfers. This is not a problem you wou
Staff Engineer
Why naively lifting scientific data to the cloud falls flat. Scientific formats predate the cloud There are exabytes of scientific data out in the wild, with more being generated every year. At Earthmover we believe the best place for it to reside is in the cloud, in object storage. Cloud platforms
Software Engineer
We at Earthmover recently released the Icechunk tensor storage engine, a novel cloud-optimized storage format and library for large-scale array data. Built on Rust’s tokio async runtime, Icechunk delivers impressive gains in performance over today’s array storage engines (e.g. Zarr V2, netCDF). The
Staff Engineer
At Earthmover, we are building the cloud platform for multidimensional array data (a.k.a. tensors). While the need for such a platform is obvious to practitioners in weather, climate, and geospatial data science (not to mention AI, where tensors reign supreme), folks from the mainstream data world o
CEO & Co-founder
Summary NASA has been migrating over 100 petabytes of data from on prem systems to the cloud for the last several years and is now able to focus on measures of efficiency that were not possible before. Earthmover and Development Seed worked with NASA to pilot Earthmover’s open-source Icechunk tensor
CEO & Co-founder
NOAA is under threat The US National Oceanic and Atmospheric Administration (NOAA) is one of the world’s leading scientific agencies. NOAA has a three part mission: 1. To understand and predict changes in climate, weather, ocean and coasts. 2. To share that knowledge and information with others. 3.
CEO & Co-founder
zarr-python’s performance paradox Last month, we released Zarr-Python 3.0 - a ground-up rewrite of the library (read more about it in this post). Beyond the exciting new features in Zarr V3, we put a lot of work into addressing some long standing performance issues with Zarr-Python 2. With the improvements described in this blog post, we’ve achieved a 14x speedup in loading the ARCO ERA5 dataset! Zarr-Python 2 had a paradoxical performance quirk; although the library could generate massive petabyte-scale datasets, it struggled to perform well when managing large or highly nested hierarchies. For example, listing the contents of a large Zarr group could be painfully slow, particularly if that Zarr group was stored on a high latency storage backend. Zarr users would experience this as long
Software Engineer
This is a blog version of a webinar that took place on August 27, 2024. Here’s a video of that webinar: Geospatial datasets representing information about real-world features such as points, lines, and polygons are increasingly large, complex, and multidimensional. They are naturally represented as
Software Engineer
This is a blog version of a webinar that took place on April 16, 2024. Here’s a video of that webinar: Earth Observation satellites generate massive volumes of data about our planet, and these data are vital for confronting global challenges. Satellite imagery is commonly distributed as individual “
CEO & Co-founder
Note: This post was originally published on the Zarr developer blog. We released Zarr-Python 2.18.0 this week. Although this release was quite light in terms of user-facing changes, it represents the beginning of a new phase for the project. In this post, we’ll walk through our plan for Zarr-Python
CEO & Co-founder
We set up a high-performance PyTorch dataloader using data stored as Zarr in the cloud Machine learning has become essential in the utilization of weather, climate, and geospatial data. Sophisticated models such as GraphCast, ClimaX, and Clay are emerging within these domains. The advancement of the
CEO & Co-founder
Earthmover is an early-stage startup building a platform for scientific data analytics in the cloud. Earthmover, an early-stage startup, is building a platform for scientific data analytics in the cloud. Our mission is to empower our customers to use scientific data to address our planet’s most urge
CEO & Co-founder
