Skip to content

module llutil.file_client

function mmcv_mkdir_or_exist

mmcv_mkdir_or_exist(dir_name, mode=511)

function has_method

has_method(obj: object, method: str) → bool

Check whether the object has a method.

Args:

  • method (str): The method name to check.

  • obj (object): The object to check.

Returns:

  • bool: True if the object has the method else False.

class BaseStorageBackend

Abstract class of storage backends.

All backends need to implement two apis: get() and get_text(). get() reads the file as a byte stream and get_text() reads the file as texts.

property name

method get

get(filepath)

method get_text

get_text(filepath)

class CephBackend

Ceph storage backend (for internal use).

Args:

  • path_mapping (dict|None): path mapping dict from local path to Petrel

  • path. When ``path_mapping={'src': 'dst'}`,srcinfilepath`

  • will be replaced bydst. Default: None.

.. warning: 

    :class:`mmcv.fileio.file_client.CephBackend` will be deprecated,
    please use :class:`mmcv.fileio.file_client.PetrelBackend` instead.




<a href="https://github.com/tjyuyao/ice-learn/blob/main/ice/llutil/file_client.py#L75"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>

### <kbd>method</kbd> `__init__`

```python
__init__(path_mapping=None)

property name

method get

get(filepath)

method get_text

get_text(filepath, encoding=None)

class PetrelBackend

Petrel storage backend (for internal use).

PetrelBackend supports reading and writing data to multiple clusters. If the file path contains the cluster name, PetrelBackend will read data from specified cluster or write data to it. Otherwise, PetrelBackend will access the default cluster.

Args:

  • path_mapping (dict, optional): Path mapping dict from local path to

  • Petrel path. When ``path_mapping={'src': 'dst'}`,src` in

  • `filepath will be replaced by dst. Default`: None.

  • enable_mc (bool, optional): Whether to enable memcached support.

  • Default: True.

Examples:

filepath1 = 's3://path/of/file'
    filepath2 = 'cluster-name:s3://path/of/file'
    client = PetrelBackend()
    client.get(filepath1)  # get data from default cluster
    client.get(filepath2)  # get data from 'cluster-name' cluster

method __init__

__init__(path_mapping: Optional[dict] = None, enable_mc: bool = True)

property name

method exists

exists(filepath: Union[str, Path]) → bool

Check whether a file path exists.

Args:

  • filepath (str or Path): Path to be checked whether exists.

Returns:

  • bool: Return True if filepath exists, False otherwise.

method get

get(filepath: Union[str, Path]) → memoryview

Read data from a given filepath with 'rb' mode.

Args:

  • filepath (str or Path): Path to read data.

Returns:

  • memoryview: A memory view of expected bytes object to avoid copying. The memoryview object can be converted to bytes by value_buf.tobytes().

method get_local_path

get_local_path(filepath: Union[str, Path]) → Iterable[str]

Download a file from filepath and return a temporary path.

get_local_path is decorated by :meth:contxtlib.contextmanager. It can be called with with statement, and when exists from the with statement, the temporary path will be released.

Args:

  • filepath (str | Path): Download a file from filepath.

Examples:

client = PetrelBackend()
    # After existing from the `with` clause,
    # the path will be removed
    with client.get_local_path('s3://path/of/your/file') as path:
#     ...     # do something here

Yields:

  • Iterable[str]: Only yield one temporary path.

method get_text

get_text(filepath: Union[str, Path], encoding: str = 'utf-8') → str

Read data from a given filepath with 'r' mode.

Args:

  • filepath (str or Path): Path to read data.

  • encoding (str): The encoding format used to open the filepath.

  • Default: 'utf-8'.

Returns:

  • str: Expected text reading from filepath.

method isdir

isdir(filepath: Union[str, Path]) → bool

Check whether a file path is a directory.

Args:

  • filepath (str or Path): Path to be checked whether it is a directory.

Returns:

  • bool: Return True if filepath points to a directory, False otherwise.

method isfile

isfile(filepath: Union[str, Path]) → bool

Check whether a file path is a file.

Args:

  • filepath (str or Path): Path to be checked whether it is a file.

Returns:

  • bool: Return True if filepath points to a file, False otherwise.

method join_path

join_path(filepath: Union[str, Path], *filepaths: Union[str, Path]) → str

Concatenate all file paths.

Args:

  • filepath (str or Path): Path to be concatenated.

Returns:

  • str: The result after concatenation.

method list_dir_or_file

list_dir_or_file(
    dir_path: Union[str, Path],
    list_dir: bool = True,
    list_file: bool = True,
    suffix: Optional[str, Tuple[str]] = None,
    recursive: bool = False
) → Iterator[str]

Scan a directory to find the interested directories or files in arbitrary order.

Note:

Petrel has no concept of directories but it simulates the directory hierarchy in the filesystem through public prefixes. In addition, if the returned path ends with '/', it means the path is a public prefix which is a logical directory.

Note:

:meth:list_dir_or_file returns the path relative to dir_path. In addition, the returned path of directory will not contains the suffix '/' which is consistent with other backends.

Args:

  • dir_path (str | Path): Path of the directory.

  • list_dir (bool): List the directories. Default: True.

  • list_file (bool): List the path of files. Default: True.

  • suffix (str or tuple[str], optional): File suffix

  • that we are interested in. Default: None.

  • recursive (bool): If set to True, recursively scan the

  • directory. Default: False.

Yields:

  • Iterable[str]: A relative path to dir_path.

method put

put(obj: bytes, filepath: Union[str, Path]) → None

Save data to a given filepath.

Args:

  • obj (bytes): Data to be saved.

  • filepath (str or Path): Path to write data.

method put_text

put_text(obj: str, filepath: Union[str, Path], encoding: str = 'utf-8') → None

Save data to a given filepath.

Args:

  • obj (str): Data to be written.

  • filepath (str or Path): Path to write data.

  • encoding (str): The encoding format used to encode the obj.

  • Default: 'utf-8'.

method remove

remove(filepath: Union[str, Path]) → None

Remove a file.

Args:

  • filepath (str or Path): Path to be removed.

class MemcachedBackend

Memcached storage backend.

Attributes:

  • server_list_cfg (str): Config file for memcached server list.

  • client_cfg (str): Config file for memcached client.

  • sys_path (str | None): Additional path to be appended to sys.path.

  • Default: None.

method __init__

__init__(server_list_cfg, client_cfg, sys_path=None)

property name

method get

get(filepath)

method get_text

get_text(filepath, encoding=None)

class LmdbBackend

Lmdb storage backend.

Args:

  • db_path (str): Lmdb database path.

  • readonly (bool, optional): Lmdb environment parameter. If True,

  • disallow any write operations. Default: True.

  • lock (bool, optional): Lmdb environment parameter. If False, when

  • concurrent access occurs, do not lock the database. Default: False.

  • readahead (bool, optional): Lmdb environment parameter. If False, disable the OS filesystem readahead mechanism, which may improve random read performance when a database is larger than RAM.

  • Default: False.

Attributes:

  • db_path (str): Lmdb database path.

method __init__

__init__(db_path, readonly=True, lock=False, readahead=False, **kwargs)

property name

method get

get(filepath)

Get values according to the filepath.

Args:

  • filepath (str | obj:Path): Here, filepath is the lmdb key.

method get_text

get_text(filepath, encoding=None)

class HardDiskBackend

Raw hard disks storage backend.

property name

method exists

exists(filepath: Union[str, Path]) → bool

Check whether a file path exists.

Args:

  • filepath (str or Path): Path to be checked whether exists.

Returns:

  • bool: Return True if filepath exists, False otherwise.

method get

get(filepath: Union[str, Path]) → bytes

Read data from a given filepath with 'rb' mode.

Args:

  • filepath (str or Path): Path to read data.

Returns:

  • bytes: Expected bytes object.

method get_local_path

get_local_path(filepath: Union[str, Path]) → Iterable[Union[str, Path]]

Only for unified API and do nothing.

method get_text

get_text(filepath: Union[str, Path], encoding: str = 'utf-8') → str

Read data from a given filepath with 'r' mode.

Args:

  • filepath (str or Path): Path to read data.

  • encoding (str): The encoding format used to open the filepath.

  • Default: 'utf-8'.

Returns:

  • str: Expected text reading from filepath.

method isdir

isdir(filepath: Union[str, Path]) → bool

Check whether a file path is a directory.

Args:

  • filepath (str or Path): Path to be checked whether it is a directory.

Returns:

  • bool: Return True if filepath points to a directory, False otherwise.

method isfile

isfile(filepath: Union[str, Path]) → bool

Check whether a file path is a file.

Args:

  • filepath (str or Path): Path to be checked whether it is a file.

Returns:

  • bool: Return True if filepath points to a file, False otherwise.

method join_path

join_path(filepath: Union[str, Path], *filepaths: Union[str, Path]) → str

Concatenate all file paths.

Join one or more filepath components intelligently. The return value is the concatenation of filepath and any members of *filepaths.

Args:

  • filepath (str or Path): Path to be concatenated.

Returns:

  • str: The result of concatenation.

method list_dir_or_file

list_dir_or_file(
    dir_path: Union[str, Path],
    list_dir: bool = True,
    list_file: bool = True,
    suffix: Optional[str, Tuple[str]] = None,
    recursive: bool = False
) → Iterator[str]

Scan a directory to find the interested directories or files in arbitrary order.

Note:

:meth:list_dir_or_file returns the path relative to dir_path.

Args:

  • dir_path (str | Path): Path of the directory.

  • list_dir (bool): List the directories. Default: True.

  • list_file (bool): List the path of files. Default: True.

  • suffix (str or tuple[str], optional): File suffix

  • that we are interested in. Default: None.

  • recursive (bool): If set to True, recursively scan the

  • directory. Default: False.

Yields:

  • Iterable[str]: A relative path to dir_path.

method put

put(obj: bytes, filepath: Union[str, Path]) → None

Write data to a given filepath with 'wb' mode.

Note:

put will create a directory if the directory of filepath does not exist.

Args:

  • obj (bytes): Data to be written.

  • filepath (str or Path): Path to write data.

method put_text

put_text(obj: str, filepath: Union[str, Path], encoding: str = 'utf-8') → None

Write data to a given filepath with 'w' mode.

Note:

put_text will create a directory if the directory of filepath does not exist.

Args:

  • obj (str): Data to be written.

  • filepath (str or Path): Path to write data.

  • encoding (str): The encoding format used to open the filepath.

  • Default: 'utf-8'.

method remove

remove(filepath: Union[str, Path]) → None

Remove a file.

Args:

  • filepath (str or Path): Path to be removed.

class HTTPBackend

HTTP and HTTPS storage bachend.

property name

method get

get(filepath)

method get_local_path

get_local_path(filepath: str) → Iterable[str]

Download a file from filepath.

get_local_path is decorated by :meth:contxtlib.contextmanager. It can be called with with statement, and when exists from the with statement, the temporary path will be released.

Args:

  • filepath (str): Download a file from filepath.

Examples:

client = HTTPBackend()
    # After existing from the `with` clause,
    # the path will be removed
    with client.get_local_path('http://path/of/your/file') as path:
#     ...     # do something here

method get_text

get_text(filepath, encoding='utf-8')

class FileClient

A general file client to access files in different backends.

The client loads a file or text in a specified backend from its path and returns it as a binary or text file. There are two ways to choose a backend, the name of backend and the prefix of path. Although both of them can be used to choose a storage backend, backend has a higher priority that is if they are all set, the storage backend will be chosen by the backend argument. If they are all None, the disk backend will be chosen. Note that It can also register other backend accessor with a given name, prefixes, and backend class. In addition, We use the singleton pattern to avoid repeated object creation. If the arguments are the same, the same object will be returned.

Args:

  • backend (str, optional): The storage backend type. Options are "disk",

  • "ceph", "memcached", "lmdb", "http" and "petrel". Default: None.

  • prefix (str, optional): The prefix of the registered storage backend.

  • Options are "s3", "http", "https". Default: None.

Examples:

# only set backend
    file_client = FileClient(backend='petrel')
    # only set prefix
    file_client = FileClient(prefix='s3')
    # set both backend and prefix but use backend to choose client
    file_client = FileClient(backend='petrel', prefix='s3')
    # if the arguments are the same, the same object is returned
    file_client1 = FileClient(backend='petrel')
    file_client1 is file_client
#     True

Attributes:

  • client (:obj:BaseStorageBackend): The backend object.

property name

method exists

exists(filepath: Union[str, Path]) → bool

Check whether a file path exists.

Args:

  • filepath (str or Path): Path to be checked whether exists.

Returns:

  • bool: Return True if filepath exists, False otherwise.

method get

get(filepath: Union[str, Path]) → Union[bytes, memoryview]

Read data from a given filepath with 'rb' mode.

Note:

There are two types of return values for get, one is bytes and the other is memoryview. The advantage of using memoryview is that you can avoid copying, and if you want to convert it to bytes, you can use .tobytes().

Args:

  • filepath (str or Path): Path to read data.

Returns:

  • bytes | memoryview: Expected bytes object or a memory view of the bytes object.

method get_local_path

get_local_path(filepath: Union[str, Path]) → Iterable[str]

Download data from filepath and write the data to local path.

get_local_path is decorated by :meth:contxtlib.contextmanager. It can be called with with statement, and when exists from the with statement, the temporary path will be released.

Note:

If the filepath is a local path, just return itself.

.. warning: 

     `get_local_path` is an experimental interface that may change in
     the future.

Args:

  • filepath (str or Path): Path to be read data.

Examples:

file_client = FileClient(prefix='s3')
    with file_client.get_local_path('s3://bucket/abc.jpg') as path:
#     ...     # do something here

Yields:

  • Iterable[str]: Only yield one path.

method get_text

get_text(filepath: Union[str, Path], encoding='utf-8') → str

Read data from a given filepath with 'r' mode.

Args:

  • filepath (str or Path): Path to read data.

  • encoding (str): The encoding format used to open the filepath.

  • Default: 'utf-8'.

Returns:

  • str: Expected text reading from filepath.

classmethod infer_client

infer_client(
    file_client_args: Optional[dict] = None,
    uri: Optional[str, Path] = None
) → FileClient

Infer a suitable file client based on the URI and arguments.

Args:

  • file_client_args (dict, optional): Arguments to instantiate a

  • FileClient. Default: None.

  • uri (str | Path, optional): Uri to be parsed that contains the file

  • prefix. Default: None.

Examples:

uri = 's3://path/of/your/file'
    file_client = FileClient.infer_client(uri=uri)
    file_client_args = {'backend': 'petrel'}
    file_client = FileClient.infer_client(file_client_args)

Returns:

  • FileClient: Instantiated FileClient object.

method isdir

isdir(filepath: Union[str, Path]) → bool

Check whether a file path is a directory.

Args:

  • filepath (str or Path): Path to be checked whether it is a directory.

Returns:

  • bool: Return True if filepath points to a directory, False otherwise.

method isfile

isfile(filepath: Union[str, Path]) → bool

Check whether a file path is a file.

Args:

  • filepath (str or Path): Path to be checked whether it is a file.

Returns:

  • bool: Return True if filepath points to a file, False otherwise.

method join_path

join_path(filepath: Union[str, Path], *filepaths: Union[str, Path]) → str

Concatenate all file paths.

Join one or more filepath components intelligently. The return value is the concatenation of filepath and any members of *filepaths.

Args:

  • filepath (str or Path): Path to be concatenated.

Returns:

  • str: The result of concatenation.

method list_dir_or_file

list_dir_or_file(
    dir_path: Union[str, Path],
    list_dir: bool = True,
    list_file: bool = True,
    suffix: Optional[str, Tuple[str]] = None,
    recursive: bool = False
) → Iterator[str]

Scan a directory to find the interested directories or files in arbitrary order.

Note:

:meth:list_dir_or_file returns the path relative to dir_path.

Args:

  • dir_path (str | Path): Path of the directory.

  • list_dir (bool): List the directories. Default: True.

  • list_file (bool): List the path of files. Default: True.

  • suffix (str or tuple[str], optional): File suffix

  • that we are interested in. Default: None.

  • recursive (bool): If set to True, recursively scan the

  • directory. Default: False.

Yields:

  • Iterable[str]: A relative path to dir_path.

method parse_uri_prefix

parse_uri_prefix(uri: Union[str, Path]) → Optional[str]

Parse the prefix of a uri.

Args:

  • uri (str | Path): Uri to be parsed that contains the file prefix.

Examples:

FileClient.parse_uri_prefix('s3://path/of/your/file')
#     's3'

Returns:

  • str | None: Return the prefix of uri if the uri contains '://' else None.

method put

put(obj: bytes, filepath: Union[str, Path]) → None

Write data to a given filepath with 'wb' mode.

Note:

put should create a directory if the directory of filepath does not exist.

Args:

  • obj (bytes): Data to be written.

  • filepath (str or Path): Path to write data.

method put_text

put_text(obj: str, filepath: Union[str, Path]) → None

Write data to a given filepath with 'w' mode.

Note:

put_text should create a directory if the directory of filepath does not exist.

Args:

  • obj (str): Data to be written.

  • filepath (str or Path): Path to write data.

  • encoding (str, optional): The encoding format used to open the

  • `filepath. Default`: 'utf-8'.

classmethod register_backend

register_backend(name, backend=None, force=False, prefixes=None)

Register a backend to FileClient.

This method can be used as a normal class method or a decorator.

.. code-block:: python

class NewBackend(BaseStorageBackend):

def get(self, filepath): return filepath

def get_text(self, filepath): return filepath

FileClient.register_backend('new', NewBackend)

or

.. code-block:: python

@FileClient.register_backend('new') class NewBackend(BaseStorageBackend):

def get(self, filepath): return filepath

def get_text(self, filepath): return filepath

Args:

  • name (str): The name of the registered backend.

  • backend (class, optional): The backend class to be registered,

  • which must be a subclass of: class:BaseStorageBackend. When this method is used as a decorator, backend is None. Defaults to None.

  • force (bool, optional): Whether to override the backend if the name has already been registered. Defaults to False.

  • prefixes (str or list[str] or tuple[str], optional): The prefixes

  • of the registered storage backend. Default: None. New in version 1.3.15.

method remove

remove(filepath: Union[str, Path]) → None

Remove a file.

Args:

  • filepath (str, Path): Path to be removed.