This class implements DataStore to interact with a stow location store. It manages a base container and can dynamically load other containers if enabled, recording them in a map. It also includes metrics for tracking operations.
Methods
CreateContainer()
@classmethod
def CreateContainer(
ctx: context.Context,
container: string
) - > stow.Container, error
Creates a new container in the stow store. This method is used to provision storage containers for data.
Parameters
| Name | Type | Description |
|---|
| ctx | context.Context | The context for the operation, allowing for cancellation and timeouts. |
| container | string | The name of the container to create. |
Returns
| Type | Description |
|---|
stow.Container, error | The newly created stow container and an error if the operation failed. |
createContainer()
@classmethod
def createContainer(
ctx: context.Context,
locID: locationID,
container: string
) - > stow.Container, error
Creates a new container in the stow store for a specific location ID. This internal method is used to provision storage containers for data.
Parameters
| Name | Type | Description |
|---|
| ctx | context.Context | The context for the operation, allowing for cancellation and timeouts. |
| locID | locationID | The identifier of the location where the container should be created. |
| container | string | The name of the container to create. |
Returns
| Type | Description |
|---|
stow.Container, error | The newly created stow container and an error if the operation failed. |
LoadContainer()
@classmethod
def LoadContainer(
ctx: context.Context,
container: string,
createIfNotFound: bool
) - > stow.Container, error
Loads an existing container from the stow store. If the container does not exist and createIfNotFound is true, it will be created.
Parameters
| Name | Type | Description |
|---|
| ctx | context.Context | The context for the operation, allowing for cancellation and timeouts. |
| container | string | The name of the container to load. |
| createIfNotFound | bool | A boolean indicating whether to create the container if it does not already exist. |
Returns
| Type | Description |
|---|
stow.Container, error | The loaded or newly created stow container and an error if the operation failed. |
loadContainer()
@classmethod
def loadContainer(
ctx: context.Context,
locID: locationID,
container: string,
createIfNotFound: bool
) - > stow.Container, error
Loads an existing container from the stow store for a specific location ID. If the container does not exist and createIfNotFound is true, it will be created.
Parameters
| Name | Type | Description |
|---|
| ctx | context.Context | The context for the operation, allowing for cancellation and timeouts. |
| locID | locationID | The identifier of the location from which to load the container. |
| container | string | The name of the container to load. |
| createIfNotFound | bool | A boolean indicating whether to create the container if it does not already exist. |
Returns
| Type | Description |
|---|
stow.Container, error | The loaded or newly created stow container and an error if the operation failed. |
getContainer()
@classmethod
def getContainer(
ctx: context.Context,
locID: locationID,
container: string
) - > c stow.Container, err error
Retrieves a container from the stow store for a specific location ID. This internal method is used to access existing containers.
Parameters
| Name | Type | Description |
|---|
| ctx | context.Context | The context for the operation, allowing for cancellation and timeouts. |
| locID | locationID | The identifier of the location from which to get the container. |
| container | string | The name of the container to retrieve. |
Returns
| Type | Description |
|---|
c stow.Container, err error | The retrieved stow container and an error if the container could not be found or accessed. |
Head()
@classmethod
def Head(
ctx: context.Context,
reference: [DataReference](datareference.md?sid=flytestdlib_storage_datareference)
) - > [Metadata](../../flyteplugins/go/tasks/pluginmachinery/catalog/metadata.md?sid=flyteplugins_go_tasks_pluginmachinery_catalog_metadata), error
Fetches metadata for a data reference without retrieving the actual data. This is useful for checking existence or properties of an object.
Parameters
| Name | Type | Description |
|---|
| ctx | context.Context | The context for the operation, allowing for cancellation and timeouts. |
| reference | [DataReference](datareference.md?sid=flytestdlib_storage_datareference) | A reference to the data object for which to retrieve metadata. |
Returns
| Type | Description |
|---|
[Metadata](../../flyteplugins/go/tasks/pluginmachinery/catalog/metadata.md?sid=flyteplugins_go_tasks_pluginmachinery_catalog_metadata), error | The metadata associated with the data reference and an error if the operation failed. |
List()
@classmethod
def List(
ctx: context.Context,
reference: [DataReference](datareference.md?sid=flytestdlib_storage_datareference),
maxItems: int,
cursor: [Cursor](cursor.md?sid=flytestdlib_storage_cursor)
) - > [][DataReference](datareference.md?sid=flytestdlib_storage_datareference), [Cursor](cursor.md?sid=flytestdlib_storage_cursor), error
Lists data references within a specified container or prefix. This allows for pagination through large sets of objects.
Parameters
| Name | Type | Description |
|---|
| ctx | context.Context | The context for the operation, allowing for cancellation and timeouts. |
| reference | [DataReference](datareference.md?sid=flytestdlib_storage_datareference) | A reference to the container or prefix within which to list items. |
| maxItems | int | The maximum number of items to return in a single listing operation. |
| cursor | [Cursor](cursor.md?sid=flytestdlib_storage_cursor) | A cursor indicating the starting point for the listing operation, used for pagination. |
Returns
| Type | Description |
|---|
[][DataReference](datareference.md?sid=flytestdlib_storage_datareference), [Cursor](cursor.md?sid=flytestdlib_storage_cursor), error | A slice of data references, a cursor for the next page of results, and an error if the operation failed. |
ReadRaw()
@classmethod
def ReadRaw(
ctx: context.Context,
reference: [DataReference](datareference.md?sid=flytestdlib_storage_datareference)
) - > io.ReadCloser, error
Reads the raw content of a data object. This method returns an io.ReadCloser for streaming the object's data.
Parameters
| Name | Type | Description |
|---|
| ctx | context.Context | The context for the operation, allowing for cancellation and timeouts. |
| reference | [DataReference](datareference.md?sid=flytestdlib_storage_datareference) | A reference to the data object to read. |
Returns
| Type | Description |
|---|
io.ReadCloser, error | An io.ReadCloser to stream the object's content and an error if the operation failed. |
WriteRaw()
@classmethod
def WriteRaw(
ctx: context.Context,
reference: [DataReference](datareference.md?sid=flytestdlib_storage_datareference),
size: int64,
opts: [Options](../../flyteplugins/go/tasks/pluginmachinery/k8s/options.md?sid=flyteplugins_go_tasks_pluginmachinery_k8s_options),
raw: io.Reader
) - > error
Writes raw content to a data object. This method allows for streaming data to a specified reference.
Parameters
| Name | Type | Description |
|---|
| ctx | context.Context | The context for the operation, allowing for cancellation and timeouts. |
| reference | [DataReference](datareference.md?sid=flytestdlib_storage_datareference) | A reference to the data object where the content will be written. |
| size | int64 | The size of the data being written in bytes. |
| opts | [Options](../../flyteplugins/go/tasks/pluginmachinery/k8s/options.md?sid=flyteplugins_go_tasks_pluginmachinery_k8s_options) | Options for the write operation, such as content type or metadata. |
| raw | io.Reader | An io.Reader providing the raw content to write. |
Returns
| Type | Description |
|---|
error | An error if the write operation failed. |
Delete()
@classmethod
def Delete(
ctx: context.Context,
reference: [DataReference](datareference.md?sid=flytestdlib_storage_datareference)
) - > error
Deletes a data object specified by its reference. This permanently removes the object from the store.
Parameters
| Name | Type | Description |
|---|
| ctx | context.Context | The context for the operation, allowing for cancellation and timeouts. |
| reference | [DataReference](datareference.md?sid=flytestdlib_storage_datareference) | A reference to the data object to delete. |
Returns
| Type | Description |
|---|
error | An error if the delete operation failed. |
GetBaseContainerFQN()
@classmethod
def GetBaseContainerFQN(
ctx: context.Context
) - > [DataReference](datareference.md?sid=flytestdlib_storage_datareference)
Retrieves the fully qualified name of the base container. This is useful for identifying the default storage location.
Parameters
| Name | Type | Description |
|---|
| ctx | context.Context | The context for the operation, allowing for cancellation and timeouts. |
Returns
| Type | Description |
|---|
[DataReference](datareference.md?sid=flytestdlib_storage_datareference) | The fully qualified name of the base container. |
CreateSignedURL()
@classmethod
def CreateSignedURL(
ctx: context.Context,
reference: [DataReference](datareference.md?sid=flytestdlib_storage_datareference),
properties: [SignedURLProperties](signedurlproperties.md?sid=flytestdlib_storage_signedurlproperties)
) - > [SignedURLResponse](signedurlresponse.md?sid=flytestdlib_storage_signedurlresponse), error
Generates a pre-signed URL for a data object, allowing temporary access without requiring direct authentication. This is useful for sharing private objects securely.
Parameters
| Name | Type | Description |
|---|
| ctx | context.Context | The context for the operation, allowing for cancellation and timeouts. |
| reference | [DataReference](datareference.md?sid=flytestdlib_storage_datareference) | A reference to the data object for which to create a signed URL. |
| properties | [SignedURLProperties](signedurlproperties.md?sid=flytestdlib_storage_signedurlproperties) | Properties for the signed URL, such as expiration time and allowed HTTP methods. |
Returns
| Type | Description |
|---|
[SignedURLResponse](signedurlresponse.md?sid=flytestdlib_storage_signedurlresponse), error | A response containing the signed URL and an error if the operation failed. |
getLocation()
@classmethod
def getLocation(
id: locationID
) - > stow.Location
Retrieves a stow location by its identifier. This internal method is used to access configured storage locations.
Parameters
| Name | Type | Description |
|---|
| id | locationID | The identifier of the location to retrieve. |
Returns
| Type | Description |
|---|
stow.Location | The stow location corresponding to the provided ID. |