# Filesystems

redun supports reading and writing to various filesystems, including local, S3, GCS, and Azure Blob Storage.

The filesystem to be used is determined by the protocol in the file URL provided to `redun.file` constructor.
In most cases, users don't need to interact with `Filesystem` objects directly.

## Azure Blob Storage filesystem

This filesystem is considered experimental - please read the documentation carefully before using it.

### Usage

If you use `redun.File` or `Dir`, initialize it with `az` path: 
`az://container@accountname.blob.core.windows.net/path/to/file`. Other paths format (like HTTPS blob URI) are not
supported.

### Thread safety

By design, `fsspec` operations are not stateless, and there are potential issues with thread safety:
 - listings cache (dircache) - for results of ls-like commands. `glob` calls *likely* don't affect the cache, 
unlike `listdir` which we don't call currently, so all redun calls like `Dir.files()` are safe 
 - file access (see https://filesystem-spec.readthedocs.io/en/latest/features.html#file-buffering-and-random-access) -
users should assume IO file operations are unsafe and avoid reads/writes to the same file from multiple redun tasks running
on the same executor. Redun gets around this issue by forcing recreation of the fsspec file system objects and restarting 
the async loop. There still may be thread safety issues, so beware.


### Credentials lookup

For most cases, Azure SDKs should work without any issues with the default credentials, and that's what
`adlfs` (`fsspec` implementation for Azure storage) does when not prompted otherwise. 

Note that it **wasn't** tested on Azure Batch nodes, but `DefaultClientCredential` is expected to work inside
any Azure compute with properly configured managed identity.

However, inside AzureML compute that uses on-behalf-of authentication, DefaultClientCredential will not work.
There is a special credential class that works, but works only in synchronous mode. `adlfs` works with both sync 
and async credential classes, but it assumes custom credential classes are async. We do a small hack in the code 
to assign credential class to a correct class property.

Currently use a simple try-catch to determine if we can retrieve auth token from DefaultAzureCredential and use
AML credential as fallback option if AML env variables are set (`AZUREML_WORKSPACE_ID`). If both options fail,
we also check if `DEFAULT_IDENTITY_CLIENT_ID` variable is set - in AzureML, it holds the client id of the cluster's
managed identity. If it is, we try to create `ManagedIdentityCredential` with it.