Expiring Data#

Over time, an Icechunk Repository will accumulate many snapshots, not all of which need to be kept around.

"Expiration" allows you to mark snapshots as expired, and "garbage collection" deletes all data (manifests, chunks, snapshots, etc.) associated with expired snapshots.

First create a Repository, configured so that there are no "inline" chunks. This will help illustrate that data is actually deleted.

import icechunk as ic

repo = ic.Repository.create(
    ic.in_memory_storage(),
    config=ic.config.RepositoryConfig(inline_chunk_threshold_bytes=0),
)

Generate a few snapshots#

Let us generate a sequence of snapshots

import zarr
import time

for i in range(10):
    session = repo.writable_session("main")
    array = zarr.create_array(
        session.store, name="array", shape=(10,), fill_value=-1, dtype=int, overwrite=True
    )
    array[:] = i
    session.commit(f"snap {i}")
    time.sleep(0.1)

There are 10 snapshots

ancestry = list(repo.ancestry(branch="main"))
print("\n\n".join([str((a.id, a.written_at)) for a in ancestry]))

('87FZPRA0B0BVR7JQ0BQG', datetime.datetime(2026, 10, 7, 15, 51, 42, 774750, tzinfo=datetime.timezone.utc))

('ZHGKHP9TZPWTB8Y4DVB0', datetime.datetime(2026, 10, 7, 15, 51, 42, 662761, tzinfo=datetime.timezone.utc))

('2505DEJX0JK0GY0X4ZQ0', datetime.datetime(2026, 10, 7, 15, 51, 42, 551243, tzinfo=datetime.timezone.utc))

('T3AZX33WDMWK1TCZ8660', datetime.datetime(2026, 10, 7, 15, 51, 42, 439791, tzinfo=datetime.timezone.utc))

('VPHM9X7PV4KFGDPEHRVG', datetime.datetime(2026, 10, 7, 15, 51, 42, 328490, tzinfo=datetime.timezone.utc))

('WNZ7AYZYTT5ST5QTE600', datetime.datetime(2026, 10, 7, 15, 51, 42, 217597, tzinfo=datetime.timezone.utc))

('419591KGFN7P4G6BYYE0', datetime.datetime(2026, 10, 7, 15, 51, 42, 105801, tzinfo=datetime.timezone.utc))

('YBWSVJMAY6V10HKCBZ7G', datetime.datetime(2026, 10, 7, 15, 51, 41, 994078, tzinfo=datetime.timezone.utc))

('V989S8R5H6CT1HWG14BG', datetime.datetime(2026, 10, 7, 15, 51, 41, 882941, tzinfo=datetime.timezone.utc))

('KMQ7XZKFJHDMAJGH6920', datetime.datetime(2026, 10, 7, 15, 51, 41, 769737, tzinfo=datetime.timezone.utc))

('1CECHNKREP0F1RSTCMT0', datetime.datetime(2026, 10, 7, 15, 51, 41, 752779, tzinfo=datetime.timezone.utc))

Expire snapshots#

Danger

Expiring snapshots is an irreversible operation. Use it with care.

First we must expire snapshots. Here we will expire any snapshot older than the 5th one.

expiry_time = ancestry[5].written_at
print(expiry_time)

2026-10-07 15:51:42.217597+00:00

expired = repo.expire_snapshots(older_than=expiry_time)
print(expired)

{'YBWSVJMAY6V10HKCBZ7G', '419591KGFN7P4G6BYYE0', 'KMQ7XZKFJHDMAJGH6920', 'V989S8R5H6CT1HWG14BG'}

This prints out the set of snapshots that were expired.

Note

The first snapshot is never expired!

The cutoff is exclusive

older_than is an exclusive bound: a snapshot is expired only if its written_at is strictly earlier than the cutoff. A snapshot whose written_at equals the cutoff is kept. The same holds for garbage_collect: an object is deleted only if it was created strictly before the cutoff and no surviving snapshot references it. This means you can pass a snapshot's own written_at as the cutoff to expire everything older than it while keeping that snapshot itself.

Confirm that these are the right snapshots (remember that ancestry list commits in decreasing order of written_at time):

print([a.id for a in ancestry[-5:-1]])

['419591KGFN7P4G6BYYE0', 'YBWSVJMAY6V10HKCBZ7G', 'V989S8R5H6CT1HWG14BG', 'KMQ7XZKFJHDMAJGH6920']

Note that ancestry is now shorter:

new_ancestry = list(repo.ancestry(branch="main"))
print("\n\n".join([str((a.id, a.written_at)) for a in new_ancestry]))

('87FZPRA0B0BVR7JQ0BQG', datetime.datetime(2026, 10, 7, 15, 51, 42, 774750, tzinfo=datetime.timezone.utc))

('ZHGKHP9TZPWTB8Y4DVB0', datetime.datetime(2026, 10, 7, 15, 51, 42, 662761, tzinfo=datetime.timezone.utc))

('2505DEJX0JK0GY0X4ZQ0', datetime.datetime(2026, 10, 7, 15, 51, 42, 551243, tzinfo=datetime.timezone.utc))

('T3AZX33WDMWK1TCZ8660', datetime.datetime(2026, 10, 7, 15, 51, 42, 439791, tzinfo=datetime.timezone.utc))

('VPHM9X7PV4KFGDPEHRVG', datetime.datetime(2026, 10, 7, 15, 51, 42, 328490, tzinfo=datetime.timezone.utc))

('WNZ7AYZYTT5ST5QTE600', datetime.datetime(2026, 10, 7, 15, 51, 42, 217597, tzinfo=datetime.timezone.utc))

('1CECHNKREP0F1RSTCMT0', datetime.datetime(2026, 10, 7, 15, 51, 41, 752779, tzinfo=datetime.timezone.utc))

Delete expired data#

Danger

Garbage collection is an irreversible operation that deletes data. Use it with care.

Use Repository.garbage_collect to delete data associated with expired snapshots

results = repo.garbage_collect(expiry_time)
print(results)

bytes_deleted: 3173 chunks_deleted: 4 manifests_deleted: 4 snapshots_deleted: 4 attributes_deleted: 0 transaction_logs_deleted: 0 objects_failed_to_delete: 0 throttled_batches: 0 delete_errors: [] skipped_phases: []

Reading the summary#

The returned GCSummary counts what was deleted, but also what wasn't:

objects_failed_to_delete counts objects whose delete request failed. They stay garbage and the next run will try again; isolated failures do not stop the run.
delete_errors holds the first few distinct error messages behind those failures.
skipped_phases lists the phases that were not run at all. Garbage collection deletes one kind of object per phase — snapshots, then transaction logs, manifests and chunks — each kind only after the kind that references it. If a phase has failures, the phases that depend on it are skipped, so a surviving object is never left pointing at something that was deleted. Those phases are simply picked up by the next run.
throttled_batches counts delete requests the store asked us to slow down. Each was retried after a pause; none of them is a failure.

A run only gives up with an error — a StorageError with kind == ErrorKind.GC_DELETES_FAILING — when deletes fail persistently, by default 50 in a row.

Tuning#

The defaults are meant to work unchanged, including on large repositories. The knobs worth knowing about, all keyword arguments of garbage_collect:

Memory. The manifest walk holds decoded manifests in memory, bounded by max_decoded_manifest_mem_bytes (4 GiB by default). Manifests grow many times over when decoded, so this, rather than the compressed budget (max_compressed_manifest_mem_bytes), is usually what bounds the memory a run uses for manifests. It does not cover the set of chunk ids the walk accumulates, which dominates on repositories with many millions of chunks.
Delete rate. Garbage collection does not delete at a fixed rate: it starts with a single request in flight and ramps up until the store throttles it or it reaches max_concurrent_deletes, then keeps adjusting. Raising that ceiling only helps on prefixes the store has already partitioned.
Listing. max_concurrent_listings sets how many object listings run concurrently while looking for garbage. It defaults to eight per available core, clamped to between 32 and 256.
Giving up. max_consecutive_delete_failures (50) is how many delete requests must fail in a row before the run aborts. Throttling only counts towards it once the store keeps throttling after the back-off between retries has grown to its maximum.

See the API reference for the full list and the exact defaults.