API Reference
Home / icechunk-python / reference
AzureCredentials
#
Credentials for an azure storage backend
This can be used to authenticate with an azure storage backend.
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
FromEnv
#
AzureStaticCredentials
#
Credentials for an azure storage backend
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
AccessKey
#
Credentials for an azure storage backend using an access key
Parameters:
Name | Type | Description | Default |
---|---|---|---|
key | str | The access key to use for authentication. | required |
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
BearerToken
#
Credentials for an azure storage backend using a bearer token
Parameters:
Name | Type | Description | Default |
---|---|---|---|
token | str | The bearer token to use for authentication. | required |
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
SasToken
#
Credentials for an azure storage backend using a shared access signature token
Parameters:
Name | Type | Description | Default |
---|---|---|---|
token | str | The shared access signature token to use for authentication. | required |
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
BasicConflictSolver
#
Bases: ConflictSolver
A basic conflict solver that allows for simple configuration of resolution behavior
This conflict solver allows for simple configuration of resolution behavior for conflicts that may occur during a rebase operation. It will attempt to resolve a limited set of conflicts based on the configuration options provided.
- When a chunk conflict is encountered, the behavior is determined by the
on_chunk_conflict
option - When an array is deleted that has been updated,
fail_on_delete_of_updated_array
will determine whether to fail the rebase operation - When a group is deleted that has been updated,
fail_on_delete_of_updated_group
will determine whether to fail the rebase operation
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
__init__(*, on_chunk_conflict=VersionSelection.UseOurs, fail_on_delete_of_updated_array=False, fail_on_delete_of_updated_group=False)
#
Create a BasicConflictSolver object with the given configuration options
Parameters:
Name | Type | Description | Default |
---|---|---|---|
on_chunk_conflict | VersionSelection | The behavior to use when a chunk conflict is encountered, by default VersionSelection.use_theirs() | UseOurs |
fail_on_delete_of_updated_array | bool | Whether to fail when a chunk is deleted that has been updated, by default False | False |
fail_on_delete_of_updated_group | bool | Whether to fail when a group is deleted that has been updated, by default False | False |
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
CachingConfig
#
Configuration for how Icechunk caches its metadata files
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 |
|
num_bytes_attributes
property
writable
#
The number of bytes of attributes to cache.
Returns:
Type | Description |
---|---|
int | None | The number of bytes of attributes to cache. |
num_bytes_chunks
property
writable
#
The number of bytes of chunks to cache.
Returns:
Type | Description |
---|---|
int | None | The number of bytes of chunks to cache. |
num_chunk_refs
property
writable
#
The number of chunk references to cache.
Returns:
Type | Description |
---|---|
int | None | The number of chunk references to cache. |
num_snapshot_nodes
property
writable
#
The number of snapshot nodes to cache.
Returns:
Type | Description |
---|---|
int | None | The number of snapshot nodes to cache. |
num_transaction_changes
property
writable
#
The number of transaction changes to cache.
Returns:
Type | Description |
---|---|
int | None | The number of transaction changes to cache. |
__init__(num_snapshot_nodes=None, num_chunk_refs=None, num_transaction_changes=None, num_bytes_attributes=None, num_bytes_chunks=None)
#
Create a new CachingConfig
object
Parameters:
Name | Type | Description | Default |
---|---|---|---|
num_snapshot_nodes | int | None | The number of snapshot nodes to cache. | None |
num_chunk_refs | int | None | The number of chunk references to cache. | None |
num_transaction_changes | int | None | The number of transaction changes to cache. | None |
num_bytes_attributes | int | None | The number of bytes of attributes to cache. | None |
num_bytes_chunks | int | None | The number of bytes of chunks to cache. | None |
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
CompressionAlgorithm
#
Bases: Enum
Enum for selecting the compression algorithm used by Icechunk to write its metadata files
Attributes:
Name | Type | Description |
---|---|---|
Zstd | int | The Zstd compression algorithm. |
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
default()
staticmethod
#
The default compression algorithm used by Icechunk to write its metadata files.
Returns:
Type | Description |
---|---|
CompressionAlgorithm | The default compression algorithm. |
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
CompressionConfig
#
Configuration for how Icechunk compresses its metadata files
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
algorithm
property
writable
#
The compression algorithm used by Icechunk to write its metadata files.
Returns:
Type | Description |
---|---|
CompressionAlgorithm | None | The compression algorithm used by Icechunk to write its metadata files. |
level
property
writable
#
The compression level used by Icechunk to write its metadata files.
Returns:
Type | Description |
---|---|
int | None | The compression level used by Icechunk to write its metadata files. |
__init__(algorithm=None, level=None)
#
Create a new CompressionConfig
object
Parameters:
Name | Type | Description | Default |
---|---|---|---|
algorithm | CompressionAlgorithm | None | The compression algorithm to use. | None |
level | int | None | The compression level to use. | None |
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
default()
staticmethod
#
The default compression configuration used by Icechunk to write its metadata files.
Returns:
Type | Description |
---|---|
CompressionConfig | |
Conflict
#
A conflict detected between snapshots
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
conflict_type
property
#
The type of conflict detected
Returns: ConflictType: The type of conflict detected
conflicted_chunks
property
#
If the conflict is a chunk conflict, this will return the list of chunk indices that are in conflict
Returns: list[list[int]] | None: The list of chunk indices that are in conflict
path
property
#
The path of the node that caused the conflict
Returns: str: The path of the node that caused the conflict
ConflictDetector
#
Bases: ConflictSolver
A conflict solver that can be used to detect conflicts between two stores, but does not resolve them
Where the BasicConflictSolver
will attempt to resolve conflicts, the ConflictDetector
will only detect them. This means that during a rebase operation the ConflictDetector
will raise a RebaseFailed
error if any conflicts are detected, and allow the rebase operation to be retried with a different conflict resolution strategy. Otherwise, if no conflicts are detected the rebase operation will succeed.
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
ConflictError
#
Bases: Exception
Error raised when a commit operation fails due to a conflict.
Source code in icechunk-python/python/icechunk/session.py
actual_parent
property
#
The actual parent snapshot ID of the branch that the session attempted to commit to.
Returns:
Type | Description |
---|---|
str | The snapshot ID of the branch tip. If this error is raised, it means the branch was modified and committed by another session after the session was created. |
expected_parent
property
#
The expected parent snapshot ID.
Returns:
Type | Description |
---|---|
str | The snapshot ID that the session was based on when the commit operation was called. |
ConflictErrorData
#
Data class for conflict errors. This describes the snapshot conflict detected when committing a session
If this error is raised, it means the branch was modified and committed by another session after the session was created.
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
actual_parent
property
#
The actual parent snapshot ID of the branch that the session attempted to commit to.
When the session is based on a branch, this is the snapshot ID of the branch tip. If this error is raised, it means the branch was modified and committed by another session after the session was created.
expected_parent
property
#
The expected parent snapshot ID.
This is the snapshot ID that the session was based on when the commit operation was called.
ConflictSolver
#
An abstract conflict solver that can be used to detect or resolve conflicts between two stores
This should never be used directly, but should be subclassed to provide specific conflict resolution behavior
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
ConflictType
#
Bases: Enum
Type of conflict detected
Attributes: NewNodeConflictsWithExistingNode: int A new node conflicts with an existing node NewNodeInInvalidGroup: tuple[int] A new node is in an invalid group ZarrMetadataDoubleUpdate: tuple[int] A zarr metadata update conflicts with an existing zarr metadata update ZarrMetadataUpdateOfDeletedArray: tuple[int] A zarr metadata update is attempted on a deleted array ZarrMetadataUpdateOfDeletedGroup: tuple[int] A zarr metadata update is attempted on a deleted group ChunkDoubleUpdate: tuple[int] A chunk update conflicts with an existing chunk update ChunksUpdatedInDeletedArray: tuple[int] Chunks are updated in a deleted array ChunksUpdatedInUpdatedArray: tuple[int] Chunks are updated in an updated array DeleteOfUpdatedArray: tuple[int] A delete is attempted on an updated array DeleteOfUpdatedGroup: tuple[int] A delete is attempted on an updated group
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
Diff
#
The result of comparing two snapshots
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
deleted_arrays
property
#
The arrays that were deleted in the target ref.
deleted_groups
property
#
The groups that were deleted in the target ref.
new_arrays
property
#
The arrays that were added to the target ref.
new_groups
property
#
The groups that were added to the target ref.
updated_chunks
property
#
The chunks that had data updated in the target ref.
updated_user_attributes
property
#
The nodes that had user attributes updated in the target ref.
updated_zarr_metadata
property
#
The nodes that had zarr metadata updated in the target ref.
GCSummary
#
Summarizes the results of a garbage collection operation on an icechunk repo
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
attributes_deleted
property
#
How many attributes were deleted.
bytes_deleted
property
#
How many bytes were deleted.
chunks_deleted
property
#
How many chunks were deleted.
manifests_deleted
property
#
How many manifests were deleted.
snapshots_deleted
property
#
How many snapshots were deleted.
transaction_logs_deleted
property
#
How many transaction logs were deleted.
GcsBearerCredential
#
Credentials for a google cloud storage backend
This is a bearer token that has an expiration time.
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
__init__(bearer, *, expires_after=None)
#
Create a GcsBearerCredential object
Parameters:
Name | Type | Description | Default |
---|---|---|---|
bearer | str | The bearer token to use for authentication. | required |
expires_after | datetime | None | The expiration time of the bearer token. | None |
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
GcsCredentials
#
Credentials for a google cloud storage backend
This can be used to authenticate with a google cloud storage backend.
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
GcsStaticCredentials
#
Credentials for a google cloud storage backend
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
ApplicationCredentials
#
Credentials for a google cloud storage backend using application default credentials
Parameters:
Name | Type | Description | Default |
---|---|---|---|
path | str | The path to the application default credentials (ADC) file. | required |
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
BearerToken
#
Credentials for a google cloud storage backend using a bearer token
Parameters:
Name | Type | Description | Default |
---|---|---|---|
token | str | The bearer token to use for authentication. | required |
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
ServiceAccount
#
Credentials for a google cloud storage backend using a service account json file
Parameters:
Name | Type | Description | Default |
---|---|---|---|
path | str | The path to the service account json file. | required |
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
ServiceAccountKey
#
Credentials for a google cloud storage backend using a a serialized service account key
Parameters:
Name | Type | Description | Default |
---|---|---|---|
key | str | The serialized service account key. | required |
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
IcechunkError
#
IcechunkStore
#
Bases: Store
, SyncMixin
Source code in icechunk-python/python/icechunk/store.py
37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 |
|
supports_listing
property
#
Does the store support listing?
supports_partial_writes
property
#
Does the store support partial writes?
supports_writes
property
#
Does the store support writes?
__init__(store, allow_pickling, *args, **kwargs)
#
Create a new IcechunkStore.
This should not be called directly, instead use the create
, open_existing
or open_or_create
class methods.
Source code in icechunk-python/python/icechunk/store.py
clear()
async
#
Clear the store.
This will remove all contents from the current session, including all groups and all arrays. But it will not modify the repository history.
Source code in icechunk-python/python/icechunk/store.py
delete(key)
async
#
Remove a key from the store
Parameters:
Name | Type | Description | Default |
---|---|---|---|
key | str | | required |
delete_dir(prefix)
async
#
Delete a prefix
Parameters:
Name | Type | Description | Default |
---|---|---|---|
prefix | str | | required |
exists(key)
async
#
Check if a key exists in the store.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
key | str | | required |
Returns:
Type | Description |
---|---|
bool | |
get(key, prototype, byte_range=None)
async
#
Retrieve the value associated with a given key.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
key | str | | required |
byte_range | ByteRequest | ByteRequest may be one of the following. If not provided, all data associated with the key is retrieved.
| None |
Returns:
Type | Description |
---|---|
Buffer | |
Source code in icechunk-python/python/icechunk/store.py
get_partial_values(prototype, key_ranges)
async
#
Retrieve possibly partial values from given key_ranges.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
key_ranges | Iterable[tuple[str, tuple[int | None, int | None]]] | Ordered set of key, range pairs, a key may occur multiple times with different ranges | required |
Returns:
Type | Description |
---|---|
list of values, in the order of the key_ranges, may contain null/none for missing keys | |
Source code in icechunk-python/python/icechunk/store.py
is_empty(prefix)
async
#
Check if the directory is empty.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
prefix | str | Prefix of keys to check. | required |
Returns:
Type | Description |
---|---|
bool | True if the store is empty, False otherwise. |
Source code in icechunk-python/python/icechunk/store.py
list()
#
Retrieve all keys in the store.
Returns:
Type | Description |
---|---|
AsyncIterator[str, None] | |
Source code in icechunk-python/python/icechunk/store.py
list_dir(prefix)
#
Retrieve all keys and prefixes with a given prefix and which do not contain the character “/” after the given prefix.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
prefix | str | | required |
Returns:
Type | Description |
---|---|
AsyncIterator[str, None] | |
Source code in icechunk-python/python/icechunk/store.py
list_prefix(prefix)
#
Retrieve all keys in the store that begin with a given prefix. Keys are returned relative to the root of the store.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
prefix | str | | required |
Returns:
Type | Description |
---|---|
AsyncIterator[str, None] | |
Source code in icechunk-python/python/icechunk/store.py
set(key, value)
async
#
Store a (key, value) pair.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
key | str | | required |
value | Buffer | | required |
Source code in icechunk-python/python/icechunk/store.py
set_if_not_exists(key, value)
async
#
Store a key to value
if the key is not already present.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
key | str | | required |
value | Buffer | | required |
Source code in icechunk-python/python/icechunk/store.py
set_partial_values(key_start_values)
async
#
Store values at a given key, starting at byte range_start.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
key_start_values | list[tuple[str, int, BytesLike]] | set of key, range_start, values triples, a key may occur multiple times with different range_starts, range_starts (considering the length of the respective values) must not specify overlapping ranges for the same key | required |
Source code in icechunk-python/python/icechunk/store.py
set_virtual_ref(key, location, *, offset, length, checksum=None, validate_container=False)
#
Store a virtual reference to a chunk.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
key | str | The chunk to store the reference under. This is the fully qualified zarr key eg: 'array/c/0/0/0' | required |
location | str | The location of the chunk in storage. This is absolute path to the chunk in storage eg: 's3://bucket/path/to/file.nc' | required |
offset | int | The offset in bytes from the start of the file location in storage the chunk starts at | required |
length | int | The length of the chunk in bytes, measured from the given offset | required |
checksum | str | datetime | None | The etag or last_medified_at field of the object | None |
validate_container | bool | If set to true, fail for locations that don't match any existing virtual chunk container | False |
Source code in icechunk-python/python/icechunk/store.py
set_virtual_refs(array_path, chunks, *, validate_containers=False)
#
Store multiple virtual references for the same array.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
array_path | str | The path to the array inside the Zarr store. Example: "/groupA/groupB/outputs/my-array" | required |
chunks | (list[VirtualChunkSpec],) | The list of virtula chunks to add | required |
validate_containers | bool | If set to true, ignore virtual references for locations that don't match any existing virtual chunk container | False |
Returns:
Type | Description |
---|---|
list[tuple[int, ...]] | None | If all virtual references where successfully updated, it returns None. If there were validation errors, it returns the chunk indices of all failed references. |
Source code in icechunk-python/python/icechunk/store.py
sync_clear()
#
Clear the store.
This will remove all contents from the current session, including all groups and all arrays. But it will not modify the repository history.
Source code in icechunk-python/python/icechunk/store.py
ManifestConfig
#
Configuration for how Icechunk manifests
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
preload
property
writable
#
The configuration for how Icechunk manifests will be preloaded.
Returns:
Type | Description |
---|---|
ManifestPreloadConfig | None | The configuration for how Icechunk manifests will be preloaded. |
__init__(preload=None)
#
Create a new ManifestConfig
object
Parameters:
Name | Type | Description | Default |
---|---|---|---|
preload | ManifestPreloadConfig | None | The configuration for how Icechunk manifests will be preloaded. | None |
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
ManifestPreloadCondition
#
Configuration for conditions under which manifests will preload on session creation
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
and_conditions(conditions)
staticmethod
#
Create a preload condition that matches only if all passed conditions
match
false()
staticmethod
#
name_matches(regex)
staticmethod
#
Create a preload condition that matches if the array's name matches the passed regex.
Example, for an array /model/outputs/temperature
, the following will match:
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
num_refs(from_refs, to_refs)
staticmethod
#
Create a preload condition that matches only if the number of chunk references in the manifest is within the given range.
from_refs is inclusive, to_refs is exclusive.
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
or_conditions(conditions)
staticmethod
#
Create a preload condition that matches if any of conditions
matches
path_matches(regex)
staticmethod
#
Create a preload condition that matches if the full path to the array matches the passed regex.
Array paths are absolute, as in /path/to/my/array
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
ManifestPreloadConfig
#
Configuration for how Icechunk manifest preload on session creation
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
max_total_refs
property
writable
#
The maximum number of references to preload.
Returns:
Type | Description |
---|---|
int | None | The maximum number of references to preload. |
preload_if
property
writable
#
The condition under which manifests will be preloaded.
Returns:
Type | Description |
---|---|
ManifestPreloadCondition | None | The condition under which manifests will be preloaded. |
__init__(max_total_refs=None, preload_if=None)
#
Create a new ManifestPreloadConfig
object
Parameters:
Name | Type | Description | Default |
---|---|---|---|
max_total_refs | int | None | The maximum number of references to preload. | None |
preload_if | ManifestPreloadCondition | None | The condition under which manifests will be preloaded. | None |
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
RebaseFailedData
#
Data class for rebase failed errors. This describes the error that occurred when rebasing a session
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
RebaseFailedError
#
Bases: Exception
Error raised when a rebase operation fails.
Source code in icechunk-python/python/icechunk/session.py
conflicts
property
#
List of conflicts that occurred during the rebase operation.
Returns:
Type | Description |
---|---|
list of Conflict | List of conflicts that occurred during the rebase operation. |
snapshot_id
property
#
The snapshot ID that the rebase operation failed on.
Returns:
Type | Description |
---|---|
str | The snapshot ID that the rebase operation failed on. |
Repository
#
An Icechunk repository.
Source code in icechunk-python/python/icechunk/repository.py
17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 |
|
config
property
#
Get a copy of this repository's config.
Returns:
Type | Description |
---|---|
RepositoryConfig | The repository configuration. |
storage
property
#
Get a copy of this repository's Storage instance.
Returns:
Type | Description |
---|---|
Storage | The repository storage instance. |
ancestry(*, branch=None, tag=None, snapshot_id=None)
#
Get the ancestry of a snapshot.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
branch | str | The branch to get the ancestry of. | None |
tag | str | The tag to get the ancestry of. | None |
snapshot_id | str | The snapshot ID to get the ancestry of. | None |
Returns:
Type | Description |
---|---|
list[SnapshotInfo] | The ancestry of the snapshot, listing out the snapshots and their metadata. |
Notes
Only one of the arguments can be specified.
Source code in icechunk-python/python/icechunk/repository.py
async_ancestry(*, branch=None, tag=None, snapshot_id=None)
#
Get the ancestry of a snapshot.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
branch | str | The branch to get the ancestry of. | None |
tag | str | The tag to get the ancestry of. | None |
snapshot_id | str | The snapshot ID to get the ancestry of. | None |
Returns:
Type | Description |
---|---|
list[SnapshotInfo] | The ancestry of the snapshot, listing out the snapshots and their metadata. |
Notes
Only one of the arguments can be specified.
Source code in icechunk-python/python/icechunk/repository.py
create(storage, config=None, virtual_chunk_credentials=None)
classmethod
#
Create a new Icechunk repository. If one already exists at the given store location, an error will be raised.
Warning
Attempting to create a Repo concurrently in the same location from multiple processes is not safe. Instead, create a Repo once and then open it concurrently.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
storage | Storage | The storage configuration for the repository. | required |
config | RepositoryConfig | The repository configuration. If not provided, a default configuration will be used. | None |
virtual_chunk_credentials | dict[str, AnyCredential] | Credentials for virtual chunks. | None |
Returns:
Type | Description |
---|---|
Self | An instance of the Repository class. |
Source code in icechunk-python/python/icechunk/repository.py
create_branch(branch, snapshot_id)
#
Create a new branch at the given snapshot.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
branch | str | The name of the branch to create. | required |
snapshot_id | str | The snapshot ID to create the branch at. | required |
Returns:
Type | Description |
---|---|
None | |
Source code in icechunk-python/python/icechunk/repository.py
create_tag(tag, snapshot_id)
#
Create a new tag at the given snapshot.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
tag | str | The name of the tag to create. | required |
snapshot_id | str | The snapshot ID to create the tag at. | required |
Returns:
Type | Description |
---|---|
None | |
Source code in icechunk-python/python/icechunk/repository.py
delete_branch(branch)
#
Delete a branch.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
branch | str | The branch to delete. | required |
Returns:
Type | Description |
---|---|
None | |
Source code in icechunk-python/python/icechunk/repository.py
delete_tag(tag)
#
Delete a tag.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
tag | str | The tag to delete. | required |
Returns:
Type | Description |
---|---|
None | |
diff(*, from_branch=None, from_tag=None, from_snapshot_id=None, to_branch=None, to_tag=None, to_snapshot_id=None)
#
Compute an overview of the operations executed from version from
to version to
.
Both versions, from
and to
, must be identified. Identification can be done using a branch, tag or snapshot id. The styles used to identify the from
and to
versions can be different.
The from
version must be a member of the ancestry
of to
.
Returns:
Type | Description |
---|---|
Diff | The operations executed between the two versions |
Source code in icechunk-python/python/icechunk/repository.py
exists(storage)
staticmethod
#
Check if a repository exists at the given storage location.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
storage | Storage | The storage configuration for the repository. | required |
Returns:
Type | Description |
---|---|
bool | True if the repository exists, False otherwise. |
Source code in icechunk-python/python/icechunk/repository.py
expire_snapshots(older_than, *, delete_expired_branches=False, delete_expired_tags=False)
#
Expire all snapshots older than a threshold.
This processes snapshots found by navigating all references in the repo, tags first, branches leter, both in lexicographical order.
Returns the ids of all snapshots considered expired and skipped from history. Notice that this snapshot are not necessarily available for garbage collection, they could still be pointed by ether refs.
If delete_expired_* is set to True, branches or tags that, after the expiration process, point to expired snapshots directly, will be deleted.
Warning: this is an administrative operation, it should be run carefully. The repository can still operate concurrently while expire_snapshots
runs, but other readers can get inconsistent views of the repository history.
Source code in icechunk-python/python/icechunk/repository.py
fetch_config(storage)
staticmethod
#
Fetch the configuration for the repository saved in storage.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
storage | Storage | The storage configuration for the repository. | required |
Returns:
Type | Description |
---|---|
RepositoryConfig | None | The repository configuration if it exists, None otherwise. |
Source code in icechunk-python/python/icechunk/repository.py
garbage_collect(delete_object_older_than)
#
Delete any objects no longer accessible from any branches or tags.
Warning: this is an administrative operation, it should be run carefully. The repository can still operate concurrently while garbage_collect
runs, but other reades can get inconsistent views if they are trying to access the expired snapshots.
Source code in icechunk-python/python/icechunk/repository.py
list_branches()
#
List the branches in the repository.
Returns:
Type | Description |
---|---|
set[str] | A set of branch names. |
list_tags()
#
List the tags in the repository.
Returns:
Type | Description |
---|---|
set[str] | A set of tag names. |
lookup_branch(branch)
#
Get the tip snapshot ID of a branch.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
branch | str | The branch to get the tip of. | required |
Returns:
Type | Description |
---|---|
str | The snapshot ID of the tip of the branch. |
Source code in icechunk-python/python/icechunk/repository.py
lookup_tag(tag)
#
Get the snapshot ID of a tag.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
tag | str | The tag to get the snapshot ID of. | required |
Returns:
Type | Description |
---|---|
str | The snapshot ID of the tag. |
Source code in icechunk-python/python/icechunk/repository.py
open(storage, config=None, virtual_chunk_credentials=None)
classmethod
#
Open an existing Icechunk repository.
If no repository exists at the given storage location, an error will be raised.
Warning
This method must be used with care in a multiprocessing context. Read more in our Parallel Write Guide.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
storage | Storage | The storage configuration for the repository. | required |
config | RepositoryConfig | The repository settings. If not provided, a default configuration will be loaded from the repository. | None |
virtual_chunk_credentials | dict[str, AnyCredential] | Credentials for virtual chunks. | None |
Returns:
Type | Description |
---|---|
Self | An instance of the Repository class. |
Source code in icechunk-python/python/icechunk/repository.py
open_or_create(storage, config=None, virtual_chunk_credentials=None)
classmethod
#
Open an existing Icechunk repository or create a new one if it does not exist.
Warning
This method must be used with care in a multiprocessing context. Read more in our Parallel Write Guide.
Attempting to create a Repo concurrently in the same location from multiple processes is not safe. Instead, create a Repo once and then open it concurrently.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
storage | Storage | The storage configuration for the repository. | required |
config | RepositoryConfig | The repository settings. If not provided, a default configuration will be loaded from the repository. | None |
virtual_chunk_credentials | dict[str, AnyCredential] | Credentials for virtual chunks. | None |
Returns:
Type | Description |
---|---|
Self | An instance of the Repository class. |
Source code in icechunk-python/python/icechunk/repository.py
readonly_session(branch=None, *, tag=None, snapshot_id=None, as_of=None)
#
Create a read-only session.
This can be thought of as a read-only checkout of the repository at a given snapshot. When branch or tag are provided, the session will be based on the tip of the branch or the snapshot ID of the tag.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
branch | str | If provided, the branch to create the session on. | None |
tag | str | If provided, the tag to create the session on. | None |
snapshot_id | str | If provided, the snapshot ID to create the session on. | None |
as_of | datetime | None | When combined with the branch argument, it will open the session at the last snapshot that is at or before this datetime | None |
Returns:
Type | Description |
---|---|
Session | The read-only session, pointing to the specified snapshot, tag, or branch. |
Notes
Only one of the arguments can be specified.
Source code in icechunk-python/python/icechunk/repository.py
reset_branch(branch, snapshot_id)
#
Reset a branch to a specific snapshot.
This will permanently alter the history of the branch such that the tip of the branch is the specified snapshot.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
branch | str | The branch to reset. | required |
snapshot_id | str | The snapshot ID to reset the branch to. | required |
Returns:
Type | Description |
---|---|
None | |
Source code in icechunk-python/python/icechunk/repository.py
save_config()
#
Save the repository configuration to storage, this configuration will be used in future calls to Repository.open.
Returns:
Type | Description |
---|---|
None | |
Source code in icechunk-python/python/icechunk/repository.py
total_chunks_storage()
#
Calculate the total storage used for chunks, in bytes .
It reports the storage needed to store all snapshots in the repository that are reachable from any branches or tags. Unreachable snapshots can be generated by using reset_branch
or expire_snapshots
. The chunks for these snapshots are not included in the result, and they should probably be deleted using garbage_collection
.
The result includes only native chunks, not adding virtual or inline chunks.
Source code in icechunk-python/python/icechunk/repository.py
writable_session(branch)
#
Create a writable session on a branch.
Like the read-only session, this can be thought of as a checkout of the repository at the tip of the branch. However, this session is writable and can be used to make changes to the repository. When ready, the changes can be committed to the branch, after which the session will become a read-only session on the new snapshot.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
branch | str | The branch to create the session on. | required |
Returns:
Type | Description |
---|---|
Session | The writable session on the branch. |
Source code in icechunk-python/python/icechunk/repository.py
RepositoryConfig
#
Configuration for an Icechunk repository
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 |
|
caching
property
writable
#
The caching configuration for the repository.
Returns:
Type | Description |
---|---|
CachingConfig | None | The caching configuration for the repository. |
compression
property
writable
#
The compression configuration for the repository.
Returns:
Type | Description |
---|---|
CompressionConfig | None | The compression configuration for the repository. |
get_partial_values_concurrency
property
writable
#
The number of concurrent requests to make when getting partial values from storage.
Returns:
Type | Description |
---|---|
int | None | The number of concurrent requests to make when getting partial values from storage. |
inline_chunk_threshold_bytes
property
writable
#
The maximum size of a chunk that will be stored inline in the repository. Chunks larger than this size will be written to storage.
manifest
property
writable
#
The manifest configuration for the repository.
Returns:
Type | Description |
---|---|
ManifestConfig | None | The manifest configuration for the repository. |
storage
property
writable
#
The storage configuration for the repository.
Returns:
Type | Description |
---|---|
StorageSettings | None | The storage configuration for the repository. |
virtual_chunk_containers
property
#
The virtual chunk containers for the repository.
Returns:
Type | Description |
---|---|
dict[str, VirtualChunkContainer] | None | The virtual chunk containers for the repository. |
__init__(inline_chunk_threshold_bytes=None, get_partial_values_concurrency=None, compression=None, caching=None, storage=None, virtual_chunk_containers=None, manifest=None)
#
Create a new RepositoryConfig
object
Parameters:
Name | Type | Description | Default |
---|---|---|---|
inline_chunk_threshold_bytes | int | None | The maximum size of a chunk that will be stored inline in the repository. | None |
get_partial_values_concurrency | int | None | The number of concurrent requests to make when getting partial values from storage. | None |
compression | CompressionConfig | None | The compression configuration for the repository. | None |
caching | CachingConfig | None | The caching configuration for the repository. | None |
storage | StorageSettings | None | The storage configuration for the repository. | None |
virtual_chunk_containers | dict[str, VirtualChunkContainer] | None | The virtual chunk containers for the repository. | None |
manifest | ManifestConfig | None | The manifest configuration for the repository. | None |
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
clear_virtual_chunk_containers()
#
default()
staticmethod
#
get_virtual_chunk_container(name)
#
Get the virtual chunk container for the repository associated with the given name.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
name | str | The name of the virtual chunk container to get. | required |
Returns:
Type | Description |
---|---|
VirtualChunkContainer | None | The virtual chunk container for the repository associated with the given name. |
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
set_virtual_chunk_container(cont)
#
Set the virtual chunk container for the repository.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
cont | VirtualChunkContainer | The virtual chunk container to set. | required |
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
S3Credentials
#
Credentials for an S3 storage backend
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
Anonymous
#
FromEnv
#
Refreshable
#
Allows for an outside authority to pass in a function that can be used to provide credentials.
This is useful for credentials that have an expiration time, or are otherwise not known ahead of time.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
pickled_function | bytes | The pickled function to use to provide credentials. | required |
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
Static
#
Uses s3 credentials without expiration
Parameters:
Name | Type | Description | Default |
---|---|---|---|
credentials | S3StaticCredentials | The credentials to use for authentication. | required |
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
S3Options
#
Options for accessing an S3-compatible storage backend
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
__init__(region=None, endpoint_url=None, allow_http=False, anonymous=False, force_path_style=False)
#
Create a new S3Options
object
Parameters:
Name | Type | Description | Default |
---|---|---|---|
region | str | None | Optional, the region to use for the storage backend. | None |
endpoint_url | str | None | Optional, the endpoint URL to use for the storage backend. | None |
allow_http | bool | Whether to allow HTTP requests to the storage backend. | False |
anonymous | bool | Whether to use anonymous credentials to the storage backend. When | False |
force_path_style | bool | Whether to force use of path-style addressing for buckets. | False |
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
S3StaticCredentials
#
Credentials for an S3 storage backend
Attributes: access_key_id: str The access key ID to use for authentication. secret_access_key: str The secret access key to use for authentication. session_token: str | None The session token to use for authentication. expires_after: datetime.datetime | None Optional, the expiration time of the credentials.
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
__init__(access_key_id, secret_access_key, session_token=None, expires_after=None)
#
Create a new S3StaticCredentials
object
Parameters:
Name | Type | Description | Default |
---|---|---|---|
access_key_id | str | The access key ID to use for authentication. | required |
secret_access_key | str | The secret access key to use for authentication. | required |
session_token | str | None | Optional, the session token to use for authentication. | None |
expires_after | datetime | None | Optional, the expiration time of the credentials. | None |
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
Session
#
A session object that allows for reading and writing data from an Icechunk repository.
Source code in icechunk-python/python/icechunk/session.py
88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 |
|
branch
property
#
The branch that the session is based on. This is only set if the session is writable.
Returns:
Type | Description |
---|---|
str or None | The branch that the session is based on if the session is writable, None otherwise. |
has_uncommitted_changes
property
#
Whether the session has uncommitted changes. This is only possibly true if the session is writable.
Returns:
Type | Description |
---|---|
bool | True if the session has uncommitted changes, False otherwise. |
read_only
property
#
Whether the session is read-only.
Returns:
Type | Description |
---|---|
bool | True if the session is read-only, False otherwise. |
snapshot_id
property
#
The base snapshot ID of the session.
Returns:
Type | Description |
---|---|
str | The base snapshot ID of the session. |
store
property
#
Get a zarr Store object for reading and writing data from the repository using zarr python.
Returns:
Type | Description |
---|---|
IcechunkStore | A zarr Store object for reading and writing data from the repository. |
all_virtual_chunk_locations()
#
Return the location URLs of all virtual chunks.
Returns:
Type | Description |
---|---|
list of str | The location URLs of all virtual chunks. |
Source code in icechunk-python/python/icechunk/session.py
allow_pickling()
#
Context manager to allow unpickling this store if writable.
Source code in icechunk-python/python/icechunk/session.py
chunk_coordinates(array_path, batch_size=1000)
async
#
Return an async iterator to all initialized chunks for the array at array_path
Returns:
Type | Description |
---|---|
an async iterator to chunk coordinates as tuples | |
Source code in icechunk-python/python/icechunk/session.py
commit(message, metadata=None)
#
Commit the changes in the session to the repository.
When successful, the writable session is completed and the session is now read-only and based on the new commit. The snapshot ID of the new commit is returned.
If the session is out of date, this will raise a ConflictError exception depicting the conflict that occurred. The session will need to be rebased before committing.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
message | str | The message to write with the commit. | required |
Returns:
Type | Description |
---|---|
str | The snapshot ID of the new commit. |
Raises:
Type | Description |
---|---|
ConflictError | If the session is out of date and a conflict occurs. |
Source code in icechunk-python/python/icechunk/session.py
discard_changes()
#
merge(other)
#
Merge the changes for this session with the changes from another session.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
other | Self | The other session to merge changes from. | required |
Source code in icechunk-python/python/icechunk/session.py
rebase(solver)
#
Rebase the session to the latest ancestry of the branch.
This method will iteratively crawl the ancestry of the branch and apply the changes from the branch to the session. If a conflict is detected, the conflict solver will be used to optionally resolve the conflict. When complete, the session will be based on the latest commit of the branch and the session will be ready to attempt another commit.
When a conflict is detected and a resolution is not possible with the provided solver, a RebaseFailed exception will be raised. This exception will contain the snapshot ID that the rebase failed on and a list of conflicts that occurred.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
solver | ConflictSolver | The conflict solver to use when a conflict is detected. | required |
Raises:
Type | Description |
---|---|
RebaseFailedError | When a conflict is detected and the solver fails to resolve it. |
Source code in icechunk-python/python/icechunk/session.py
SnapshotInfo
#
Metadata for a snapshot
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
Storage
#
Storage configuration for an IcechunkStore
Currently supports memory, filesystem S3, azure blob, and google cloud storage backends. Use the following methods to create a Storage object with the desired backend.
Ex:
storage = icechunk.in_memory_storage()
storage = icechunk.local_filesystem_storage("/path/to/root")
storage = icechunk.s3_storage("bucket", "prefix", ...)
storage = icechunk.gcs_storage("bucket", "prefix", ...)
storage = icechunk.azure_storage("container", "prefix", ...)
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 |
|
StorageConcurrencySettings
#
Configuration for how Icechunk uses its Storage instance
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
ideal_concurrent_request_size
property
writable
#
The ideal concurrent request size.
Returns:
Type | Description |
---|---|
int | None | The ideal concurrent request size. |
max_concurrent_requests_for_object
property
writable
#
The maximum number of concurrent requests for an object.
Returns:
Type | Description |
---|---|
int | None | The maximum number of concurrent requests for an object. |
__init__(max_concurrent_requests_for_object=None, ideal_concurrent_request_size=None)
#
Create a new StorageConcurrencySettings
object
Parameters:
Name | Type | Description | Default |
---|---|---|---|
max_concurrent_requests_for_object | int | None | The maximum number of concurrent requests for an object. | None |
ideal_concurrent_request_size | int | None | The ideal concurrent request size. | None |
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
StorageSettings
#
Configuration for how Icechunk uses its Storage instance
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
concurrency
property
#
The configuration for how much concurrency Icechunk store uses
Returns:
Type | Description |
---|---|
StorageConcurrencySettings | None | The configuration for how Icechunk uses its Storage instance. |
unsafe_use_conditional_create
property
#
True if Icechunk will use conditional PUT operations for creation in the object store
unsafe_use_conditional_update
property
#
True if Icechunk will use conditional PUT operations for updates in the object store
unsafe_use_metadata
property
#
True if Icechunk will write object metadata in the object store
__init__(concurrency=None, unsafe_use_conditional_create=None, unsafe_use_conditional_update=None, unsafe_use_metadata=None)
#
Create a new StorageSettings
object
Parameters:
Name | Type | Description | Default |
---|---|---|---|
concurrency | StorageConcurrencySettings | None | The configuration for how Icechunk uses its Storage instance. | None |
unsafe_use_conditional_update | bool | None | If set to False, Icechunk loses some of its consistency guarantees. This is only useful in object stores that don't support the feature. Use it at your own risk. | None |
unsafe_use_conditional_create | bool | None | If set to False, Icechunk loses some of its consistency guarantees. This is only useful in object stores that don't support the feature. Use at your own risk. | None |
unsafe_use_metadata | bool | None | Don't write metadata fields in Icechunk files. This is only useful in object stores that don't support the feature. Use at your own risk. | None |
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
VersionSelection
#
Bases: Enum
Enum for selecting the which version of a conflict
Attributes:
Name | Type | Description |
---|---|---|
Fail | int | Fail the rebase operation |
UseOurs | int | Use the version from the source store |
UseTheirs | int | Use the version from the target store |
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
VirtualChunkContainer
#
A virtual chunk container is a configuration that allows Icechunk to read virtual references from a storage backend.
Attributes:
Name | Type | Description |
---|---|---|
name | str | The name of the virtual chunk container. |
url_prefix | str | The prefix of urls that will use this containers configuration for reading virtual references. |
store | ObjectStoreConfig | The storage backend to use for the virtual chunk container. |
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
__init__(name, url_prefix, store)
#
Create a new VirtualChunkContainer
object
Parameters:
Name | Type | Description | Default |
---|---|---|---|
name | str | The name of the virtual chunk container. | required |
url_prefix | str | The prefix of urls that will use this containers configuration for reading virtual references. | required |
store | AnyObjectStoreConfig | The storage backend to use for the virtual chunk container. | required |
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
VirtualChunkSpec
#
The specification for a virtual chunk reference.
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
etag_checksum
property
#
Optional object store e-tag for the containing object.
Icechunk will refuse to serve data from this chunk if the etag has changed.
index
property
#
The chunk index, in chunk coordinates space
last_updated_at_checksum
property
#
Optional timestamp for the containing object.
Icechunk will refuse to serve data from this chunk if it has been modified in object store after this time.
length
property
#
The length of the chunk in bytes
location
property
#
The URL to the virtual chunk data, something like 's3://bucket/foo.nc'
offset
property
#
The chunk offset within the pointed object, in bytes
azure_credentials(*, access_key=None, sas_token=None, bearer_token=None, from_env=None)
#
Create credentials Azure Blob Storage object store.
If all arguments are None, credentials are fetched from the operative system environment.
Source code in icechunk-python/python/icechunk/credentials.py
azure_from_env_credentials()
#
Instruct Azure Blob Storage object store to fetch credentials from the operative system environment.
azure_static_credentials(*, access_key=None, sas_token=None, bearer_token=None)
#
Create static credentials Azure Blob Storage object store.
Source code in icechunk-python/python/icechunk/credentials.py
azure_storage(*, account, container, prefix, access_key=None, sas_token=None, bearer_token=None, from_env=None, config=None)
#
Create a Storage instance that saves data in Azure Blob Storage object store.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
account | str | The account to which the caller must have access privileges | required |
container | str | The container where the repository will store its data | required |
prefix | str | The prefix within the container that is the root directory of the repository | required |
access_key | str | None | Azure Blob Storage credential access key | None |
sas_token | str | None | Azure Blob Storage credential SAS token | None |
bearer_token | str | None | Azure Blob Storage credential bearer token | None |
from_env | bool | None | Fetch credentials from the operative system environment | None |
Source code in icechunk-python/python/icechunk/storage.py
containers_credentials(m={}, **kwargs)
#
Build a map of credentials for virtual chunk containers.
Example usage:
import icechunk as ic
config = ic.RepositoryConfig.default()
config.inline_chunk_threshold_bytes = 512
virtual_store_config = ic.s3_store(
region="us-east-1",
endpoint_url="http://localhost:9000",
allow_http=True,
s3_compatible=True,
force_path_style=True,
)
container = ic.VirtualChunkContainer("s3", "s3://", virtual_store_config)
config.set_virtual_chunk_container(container)
credentials = ic.containers_credentials(
s3=ic.s3_credentials(access_key_id="ACCESS_KEY", secret_access_key="SECRET")
)
repo = ic.Repository.create(
storage=ic.local_filesystem_storage(store_path),
config=config,
virtual_chunk_credentials=credentials,
)
Parameters:
Name | Type | Description | Default |
---|---|---|---|
m | Mapping[str, AnyS3Credential] | A mapping from container name to credentials. | {} |
Source code in icechunk-python/python/icechunk/credentials.py
gcs_credentials(*, service_account_file=None, service_account_key=None, application_credentials=None, bearer_token=None, from_env=None, get_credentials=None)
#
Create credentials Google Cloud Storage object store.
If all arguments are None, credentials are fetched from the operative system environment.
Source code in icechunk-python/python/icechunk/credentials.py
gcs_from_env_credentials()
#
Instruct Google Cloud Storage object store to fetch credentials from the operative system environment.
gcs_refreshable_credentials(get_credentials)
#
Create refreshable credentials for Google Cloud Storage object store.
Source code in icechunk-python/python/icechunk/credentials.py
gcs_static_credentials(*, service_account_file=None, service_account_key=None, application_credentials=None, bearer_token=None)
#
Create static credentials Google Cloud Storage object store.
Source code in icechunk-python/python/icechunk/credentials.py
gcs_storage(*, bucket, prefix, service_account_file=None, service_account_key=None, application_credentials=None, bearer_token=None, from_env=None, config=None, get_credentials=None)
#
Create a Storage instance that saves data in Google Cloud Storage object store.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
bucket | str | The bucket where the repository will store its data | required |
prefix | str | None | The prefix within the bucket that is the root directory of the repository | required |
from_env | bool | None | Fetch credentials from the operative system environment | None |
bearer_token | str | None | The bearer token to use for the object store | None |
get_credentials | Callable[[], GcsBearerCredential] | None | Use this function to get and refresh object store credentials | None |
Source code in icechunk-python/python/icechunk/storage.py
in_memory_storage()
#
Create a Storage instance that saves data in memory.
This Storage implementation is used for tests. Data will be lost after the process finishes, and can only be accesses through the Storage instance returned. Different instances don't share data.
Source code in icechunk-python/python/icechunk/storage.py
initialize_logs()
#
Initialize the logging system for the library.
This should be called before any other Icechunk functions are called.
local_filesystem_storage(path)
#
Create a Storage instance that saves data in the local file system.
This Storage instance is not recommended for production data
Source code in icechunk-python/python/icechunk/storage.py
r2_storage(*, bucket=None, prefix=None, account_id=None, endpoint_url=None, region=None, allow_http=False, access_key_id=None, secret_access_key=None, session_token=None, expires_after=None, anonymous=None, from_env=None, get_credentials=None)
#
Create a Storage instance that saves data in Tigris object store.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
bucket | str | None | The bucket name | None |
prefix | str | None | The prefix within the bucket that is the root directory of the repository | None |
account_id | str | None | Cloudflare account ID. When provided, a default endpoint URL is constructed as | None |
endpoint_url | str | None | Endpoint where the object store serves data, example: | None |
region | str | None | The region to use in the object store, if | None |
allow_http | bool | If the object store can be accessed using http protocol instead of https | False |
access_key_id | str | None | S3 credential access key | None |
secret_access_key | str | None | S3 credential secret access key | None |
session_token | str | None | Optional S3 credential session token | None |
expires_after | datetime | None | Optional expiration for the object store credentials | None |
anonymous | bool | None | If set to True requests to the object store will not be signed | None |
from_env | bool | None | Fetch credentials from the operative system environment | None |
get_credentials | Callable[[], S3StaticCredentials] | None | Use this function to get and refresh object store credentials | None |
Source code in icechunk-python/python/icechunk/storage.py
s3_anonymous_credentials()
#
Create no-signature credentials for S3 and S3 compatible object stores.
s3_credentials(*, access_key_id=None, secret_access_key=None, session_token=None, expires_after=None, anonymous=None, from_env=None, get_credentials=None)
#
Create credentials for S3 and S3 compatible object stores.
If all arguments are None, credentials are fetched from the environment.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
access_key_id | str | None | S3 credential access key | None |
secret_access_key | str | None | S3 credential secret access key | None |
session_token | str | None | Optional S3 credential session token | None |
expires_after | datetime | None | Optional expiration for the object store credentials | None |
anonymous | bool | None | If set to True requests to the object store will not be signed | None |
from_env | bool | None | Fetch credentials from the operative system environment | None |
get_credentials | Callable[[], S3StaticCredentials] | None | Use this function to get and refresh object store credentials | None |
Source code in icechunk-python/python/icechunk/credentials.py
100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 |
|
s3_from_env_credentials()
#
Instruct S3 and S3 compatible object stores to gather credentials from the operative system environment.
s3_refreshable_credentials(get_credentials)
#
Create refreshable credentials for S3 and S3 compatible object stores.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
get_credentials | Callable[[], S3StaticCredentials] | Use this function to get and refresh the credentials. The function must be pickable. | required |
Source code in icechunk-python/python/icechunk/credentials.py
s3_static_credentials(*, access_key_id, secret_access_key, session_token=None, expires_after=None)
#
Create static credentials for S3 and S3 compatible object stores.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
access_key_id | str | S3 credential access key | required |
secret_access_key | str | S3 credential secret access key | required |
session_token | str | None | Optional S3 credential session token | None |
expires_after | datetime | None | Optional expiration for the object store credentials | None |
Source code in icechunk-python/python/icechunk/credentials.py
s3_storage(*, bucket, prefix, region=None, endpoint_url=None, allow_http=False, access_key_id=None, secret_access_key=None, session_token=None, expires_after=None, anonymous=None, from_env=None, get_credentials=None, force_path_style=False)
#
Create a Storage instance that saves data in S3 or S3 compatible object stores.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
bucket | str | The bucket where the repository will store its data | required |
prefix | str | None | The prefix within the bucket that is the root directory of the repository | required |
region | str | None | The region to use in the object store, if | None |
endpoint_url | str | None | Optional endpoint where the object store serves data, example: http://localhost:9000 | None |
allow_http | bool | If the object store can be accessed using http protocol instead of https | False |
access_key_id | str | None | S3 credential access key | None |
secret_access_key | str | None | S3 credential secret access key | None |
session_token | str | None | Optional S3 credential session token | None |
expires_after | datetime | None | Optional expiration for the object store credentials | None |
anonymous | bool | None | If set to True requests to the object store will not be signed | None |
from_env | bool | None | Fetch credentials from the operative system environment | None |
get_credentials | Callable[[], S3StaticCredentials] | None | Use this function to get and refresh object store credentials | None |
force_path_style | bool | Whether to force using path-style addressing for buckets | False |
Source code in icechunk-python/python/icechunk/storage.py
s3_store(region=None, endpoint_url=None, allow_http=False, anonymous=False, s3_compatible=False, force_path_style=False)
#
Build an ObjectStoreConfig instance for S3 or S3 compatible object stores.
Source code in icechunk-python/python/icechunk/storage.py
spec_version()
#
The version of the Icechunk specification that the library is compatible with.
Returns: int: The version of the Icechunk specification that the library is compatible with
Source code in icechunk-python/python/icechunk/_icechunk_python.pyi
tigris_storage(*, bucket, prefix, region=None, endpoint_url=None, use_weak_consistency=False, allow_http=False, access_key_id=None, secret_access_key=None, session_token=None, expires_after=None, anonymous=None, from_env=None, get_credentials=None)
#
Create a Storage instance that saves data in Tigris object store.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
bucket | str | The bucket where the repository will store its data | required |
prefix | str | None | The prefix within the bucket that is the root directory of the repository | required |
region | str | None | The region to use in the object store, if | None |
endpoint_url | str | None | Optional endpoint where the object store serves data, example: http://localhost:9000 | None |
use_weak_consistency | bool | If set to True it will return a Storage instance that is read only, and can read from the the closest Tigris region. Behavior is undefined if objects haven't propagated to the region yet. This option is for experts only. | False |
allow_http | bool | If the object store can be accessed using http protocol instead of https | False |
access_key_id | str | None | S3 credential access key | None |
secret_access_key | str | None | S3 credential secret access key | None |
session_token | str | None | Optional S3 credential session token | None |
expires_after | datetime | None | Optional expiration for the object store credentials | None |
anonymous | bool | None | If set to True requests to the object store will not be signed | None |
from_env | bool | None | Fetch credentials from the operative system environment | None |
get_credentials | Callable[[], S3StaticCredentials] | None | Use this function to get and refresh object store credentials | None |
Source code in icechunk-python/python/icechunk/storage.py
XarrayDatasetWriter
dataclass
#
Write Xarray Datasets to a group in an Icechunk store.
This class is private API. Please do not use it.
Source code in icechunk-python/python/icechunk/xarray.py
71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 |
|
write_eager()
#
Write in-memory variables to store.
Returns:
Type | Description |
---|---|
None | |
Source code in icechunk-python/python/icechunk/xarray.py
write_lazy(chunkmanager_store_kwargs=None, split_every=None)
#
Write lazy arrays (e.g. dask) to store.
Source code in icechunk-python/python/icechunk/xarray.py
write_metadata(encoding=None)
#
This method creates new Zarr arrays when necessary, writes attributes, and any in-memory arrays.
Source code in icechunk-python/python/icechunk/xarray.py
make_dataset(obj)
#
Copied from DataArray.to_zarr
Source code in icechunk-python/python/icechunk/xarray.py
to_icechunk(obj, session, *, group=None, mode=None, safe_chunks=True, append_dim=None, region=None, encoding=None, chunkmanager_store_kwargs=None, split_every=None)
#
Write an Xarray object to a group of an Icechunk store.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
obj | DataArray | Dataset | Xarray object to write | required |
session | Session | Writable Icechunk Session | required |
mode | "w", "w-", "a", "a-", r+", None | Persistence mode: "w" means create (overwrite if exists); "w-" means create (fail if exists); "a" means override all existing variables including dimension coordinates (create if does not exist); "a-" means only append those variables that have | "w" |
group | str | Group path. (a.k.a. | None |
encoding | dict | Nested dictionary with variable names as keys and dictionaries of variable specific encodings as values, e.g., | None |
append_dim | hashable | If set, the dimension along which the data will be appended. All other dimensions on overridden variables must remain the same size. | None |
region | dict or auto | Optional mapping from dimension names to either a) If Alternatively integer slices can be provided; for example, Users are expected to ensure that the specified region aligns with Zarr chunk boundaries, and that dask chunks are also aligned. Xarray makes limited checks that these multiple chunk boundaries line up. It is possible to write incomplete chunks and corrupt the data with this option if you are not careful. | None |
safe_chunks | bool | If True, only allow writes to when there is a many-to-one relationship between Zarr chunks (specified in encoding) and Dask chunks. Set False to override this restriction; however, data may become corrupted if Zarr arrays are written in parallel. In addition to the many-to-one relationship validation, it also detects partial chunks writes when using the region parameter, these partial chunks are considered unsafe in the mode "r+" but safe in the mode "a". Note: Even with these validations it can still be unsafe to write two or more chunked arrays in the same location in parallel if they are not writing in independent regions. | True |
chunkmanager_store_kwargs | dict | Additional keyword arguments passed on to the | None |
split_every | int | None | Number of tasks to merge at every level of the tree reduction. | None |
Returns:
Type | Description |
---|---|
None | |
Notes
Two restrictions apply to the use of region
:
- If
region
is set, all variables in a dataset must have at least one dimension in common with the region. Other variables should be written in a separate single call toto_icechunk()
. - Dimensions cannot be included in both
region
andappend_dim
at the same time. To create empty arrays to fill in withregion
, use theXarrayDatasetWriter
directly.
Source code in icechunk-python/python/icechunk/xarray.py
192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 |
|