Skip to content

Icechunk

Open-source, high-performance array format and library for scientific data

Python Rust JavaScript R Java Julia More →
Read the docs
Scale and Speed Gigabytes to petabytes. Never the bottleneck. Zero-Copy Ingest HDF5, NetCDF, h5ad, TIFF, ND2, GRIB and more Versioning Cite exact versions and collaborate on branches ACID Transactions Commits are all or nothing. No partial reads. No Server Required Client and format only. No service to run.

Home

Scale and Speed#

Icechunk works the same whether you store gigabytes on a laptop, terabytes in a bucket, or reference petabytes of existing files from a repository of a few gigabytes.

Reads stay fast at every size: Icechunk fetches only the chunks you need and downloads them in parallel.

Individual researchers use Icechunk, and government agencies are adopting it for national-scale projects. NOAA's National Weather Service has adopted it for its CIRRUS cloud data platform (blog post), and a NASA feasibility report recommends it for virtual data stores (report).

Tuning performance · Parallel writes

Zero-Copy Ingest#

Reading a TIFF, HDF5, NetCDF or GRIB file from the cloud is slow. A reader has to fetch headers and offset tables before it finds the bytes it wants. Each lookup is another round trip, so it is often faster to download the entire file first.

With Icechunk, you parse each file once and record where every chunk's bytes are. Every reader after that skips the parsing. Reading a virtual chunk (a pointer to bytes in the original file) is one range request instead of many round trips. This makes the files cloud-optimized without duplicating or changing the originals.

This technique turned the 115 TB GOES-16 satellite archive (380,000 NetCDF files) into an Icechunk repository of about 80 GB (blog post).

Virtual datasets

Mix Virtual and Native Chunks#

You don't have to choose between virtual and native chunks. One repository, and even one array, can mix both. Reference an archive where it sits, then append new data, fix a bad chunk, or add downsampled levels as native chunks. Anyone reading the repository sees ordinary Zarr and never needs to know which kind of chunk they are reading.

Versioning#

In a plain Zarr store, a bad write or an accidental delete is permanent. Icechunk records every change as a commit, so you can undo a mistake by moving the branch back to an earlier commit. Old commits stay readable until you choose to expire them.

Tag a commit to cite the exact data in a paper, or create a branch to try a change without affecting anyone else.

Version control

createadd Januaryadd Februaryfix Februarytry a new maskmaintag: v1experiment createadd Januaryadd Februaryfix Februarytry a new masktag: v1mainexperiment

History Without Ballooning Storage#

Keeping history doesn't mean keeping full copies. A commit writes only the chunks that changed, and every other chunk points at data already in storage.

Expiring old data

ACID Transactions#

Without transactions, a reader can catch a write halfway. The result is a Schrödinger's array, but unlike Schrödinger's cat, looking at it doesn't settle it into one state. You can get some chunks from before the write and some from after, an array that never existed. The animation shows a reader catching one.

With Icechunk, a reader always sees one complete commit, even while another process is writing. When two writers commit at once, the second is rejected or rebased, never silently overwritten.

Transactions

No Server Required#

The Icechunk library runs inside your code and reads and writes files directly. There is no server, database or service to run.

created with by