secretflow.data.io#

secretflow.data.io.oss#

Functions:

s3fs()

Return a s3 filesystem instance.

open(path[, mode])

Open a oss object.
secretflow.data.io.oss.s3fs()[source]#

Return a s3 filesystem instance.

secretflow.data.io.oss.open(path, mode='rb')[source]#

Open a oss object.

Parameters:

  • path – oss file path.

  • mode – optional; open mode.

Returns:

A file-like object.

secretflow.data.io.util#

Functions:

open(filepath[, mode])

Open a oss/http/https file.

is_local_file(uri)

read_csv_wrapper(filepath[, ...])

A wrapper of pandas read_csv and supports oss file.

to_csv_wrapper(df, filepath, **kwargs)

A wrapper of pandas to_csv and supports oss file.
secretflow.data.io.util.open(filepath: Union[str, Path], mode='rb')[source]#

Open a oss/http/https file.

Parameters:

  • filepath – The file path, which can be an oss, or pathlib.Path object.

  • mode – optional. open mode.

Returns:

the file object.

secretflow.data.io.util.is_local_file(uri: str) → bool[source]#
secretflow.data.io.util.read_csv_wrapper(filepath: str, auto_gen_header_prefix: str = '', **kwargs) → DataFrame[source]#

A wrapper of pandas read_csv and supports oss file.

Parameters:

  • filepath – the file path.

  • auto_gen_header_prefix – If set, the format of generated headers would be {gen_header_prefix}_{col_idx}.

  • kwargs – all other arguments are same with pandas.DataFrame.read_csv().

Returns:

a pandas DataFrame.

secretflow.data.io.util.to_csv_wrapper(df: DataFrame, filepath, **kwargs)[source]#

A wrapper of pandas to_csv and supports oss file.

Parameters:

  • filepath – the file path.

  • kwargs – all other arguments are same with pandas.DataFrame.read_csv().

Returns:

a pandas DataFrame.