Flaky Networks#

Icechunk provides configurable retries and timeouts to handle unreliable networks.

Note

If you encounter errors or operations that you believe should be retried but aren't, please open an issue.

Retries: `StorageRetriesSettings`#

Controls retry behavior for all storage operations (reads, writes, deletes).

Parameter	Default	Description
`max_tries`	10	Maximum number of attempts (including the initial request)
`initial_backoff_ms`	100	Initial delay (ms) for the exponential backoff stage
`max_backoff_ms`	180,000 (3 min)	Maximum delay (ms) between retries

import icechunk

config = icechunk.RepositoryConfig.default()
config.storage.retries = icechunk.StorageRetriesSettings(
    max_tries=20,
    initial_backoff_ms=200,
    max_backoff_ms=60_000,
)

Timeouts: `StorageTimeoutSettings`#

Controls connect, read, and operation timeouts for storage requests. All values are in milliseconds. A value of 0 disables the timeout, including the default. A request that times out is retried according to the retry settings.

Parameter	S3 default	`object_store` default	Description
`connect_timeout_ms`	3.1 s	5 s	Time allowed to establish a TCP connection
`read_timeout_ms`	none	not supported	Time from sending a request, including its body, until the response headers arrive
`operation_timeout_ms`	none	not supported	Total time allowed for an entire operation (including all retries)
`operation_attempt_timeout_ms`	10 min	30 s	Time allowed for a single attempt of an operation

The S3 column applies to s3_storage, r2_storage, tigris_storage and hf_storage. The object_store column applies to gcs_storage, azure_storage, http_storage and s3_object_store_storage. For these, the values set here take precedence over the same options passed in the backend config.

The per-attempt timeout includes uploading the request body. Concurrent requests share the network, so with many concurrent writes on a slow connection a single upload can take much longer than its size alone suggests. Set the per-attempt timeout to allow for that. For the S3 client the per-attempt timeout does not include downloading a response body; the stalled stream detection below covers that. For the object_store backends it does.

config.storage.timeouts = icechunk.StorageTimeoutSettings(
    connect_timeout_ms=5_000,
    read_timeout_ms=30_000,
    operation_timeout_ms=300_000,
    operation_attempt_timeout_ms=60_000,
)

Stalled stream detection#

A stalled network stream is an HTTP connection which does not transfer any data over a certain period. Stalled connections may occur in the following situations:

When the client is connecting to a remote object store behind a slow network connection.
When the client is behind a VPN or proxy server which is limiting the number or throughput of connections between the client and the remote object store.

The Amazon S3 SDK allows detection and retrying on stalled stream errors, which typically look like

  |-> I/O error
  |-> streaming error
  `-> minimum throughput was specified at 1 B/s, but throughput of 0 B/s was observed

This behavior is configurable when creating a new Storage option, via the network_stream_timeout_seconds parameter. The default is 60 seconds. To set a different value, you may specify as follows

storage = icechunk.s3_storage(
    **other_storage_kwargs,
    network_stream_timeout_seconds=50,
)
repo = icechunk.Repository.open(storage=storage)

Specifying a value of 0 disables this check entirely.

Applying the configuration#

Pass the configured RepositoryConfig when creating or opening a repository:

repo = icechunk.Repository.open(
    storage=storage,
    config=config,
)

See Configuration for full details on applying and persisting configuration.