secretflow.utils#

secretflow.utils.simulation

secretflow.utils.cloudpickle#

This file mainly borrows from ray.cloudpickle.cloudpickle_fast.py.

We introduce a custom serialization omitting code positions, e.g., function filename, first line no, .etc.

It is useful for some scenarios that need to generate the signature of a function deterministicly and is used to sign python functions to execute in TEE.

Classes:

CodePositionIndependentCloudPickler(file[, ...])

Functions:

code_position_independent_dumps(obj[, protocol])

Serialize obj as a string of bytes allocated in memory.

class secretflow.utils.cloudpickle.CodePositionIndependentCloudPickler(file, protocol=None, buffer_callback=None)[源代码]#

基类：CloudPickler

Attributes:

dispatch_table

Methods:

`dump`(obj)	Write a pickled representation of the given object to the open file.
`__init__`(file[, protocol, buffer_callback])
`reducer_override`(obj)	Type-agnostic reducing callback for function and classes.

dispatch_table = ChainMap({<class 'classmethod'>: <function _classmethod_reduce>, <class '_io.TextIOWrapper'>: <function _file_reduce>, <class 'logging.Logger'>: <function _logger_reduce>, <class 'logging.RootLogger'>: <function _root_logger_reduce>, <class 'memoryview'>: <function _memoryview_reduce>, <class 'property'>: <function _property_reduce>, <class 'staticmethod'>: <function _classmethod_reduce>, <class 'cell'>: <function _cell_reduce>, <class 'code'>: <function _code_reduce>, <class 'getset_descriptor'>: <function _getset_descriptor_reduce>, <class 'module'>: <function _module_reduce>, <class 'method'>: <function _method_reduce>, <class 'mappingproxy'>: <function _mappingproxy_reduce>, <class '_weakrefset.WeakSet'>: <function _weakset_reduce>, <class 'typing.TypeVar'>: <function _typevar_reduce>, <class 'dict_keys'>: <function _dict_keys_reduce>, <class 'dict_values'>: <function _dict_values_reduce>, <class 'dict_items'>: <function _dict_items_reduce>, <class 'odict_keys'>: <function _odict_keys_reduce>, <class 'odict_values'>: <function _odict_values_reduce>, <class 'odict_items'>: <function _odict_items_reduce>}, {<class 'complex'>: <function pickle_complex>, <class 're.Pattern'>: <function _pickle>, <class 'soupsieve.css_types.Selector'>: <function _pickle>, <class 'soupsieve.css_types.SelectorNull'>: <function _pickle>, <class 'soupsieve.css_types.SelectorTag'>: <function _pickle>, <class 'soupsieve.css_types.SelectorAttribute'>: <function _pickle>, <class 'soupsieve.css_types.SelectorContains'>: <function _pickle>, <class 'soupsieve.css_types.SelectorNth'>: <function _pickle>, <class 'soupsieve.css_types.SelectorLang'>: <function _pickle>, <class 'soupsieve.css_types.SelectorList'>: <function _pickle>, <class 'soupsieve.css_match.SoupSieve'>: <function _pickle>, <class 'numpy.ufunc'>: <function _ufunc_reduce>, <class 'numpy._DTypeMeta'>: <function _DType_reduce>, <class 'uarray._Function'>: <function pickle_function>, <class 'uarray._BackendState'>: <function pickle_state>, <class 'uarray._SetBackendContext'>: <function pickle_set_backend_context>, <class 'uarray._SkipBackendContext'>: <function pickle_skip_backend_context>, <class 'torch.layout'>: <function <lambda>>})#

dump(obj)[源代码]#: Write a pickled representation of the given object to the open file.

__init__(file, protocol=None, buffer_callback=None)[源代码]#

reducer_override(obj)[源代码]#

Type-agnostic reducing callback for function and classes.

For performance reasons, subclasses of the C _pickle.Pickler class cannot register custom reducers for functions and classes in the dispatch_table. Reducer for such types must instead implemented in the special reducer_override method.

Note that method will be called for any object except a few builtin-types (int, lists, dicts etc.), which differs from reducers in the Pickler’s dispatch_table, each of them being invoked for objects of a specific type only.

This property comes in handy for classes: although most classes are instances of the type metaclass, some of them can be instances of other custom metaclasses (such as enum.EnumMeta for example). In particular, the metaclass will likely not be known in advance, and thus cannot be special-cased using an entry in the dispatch_table. reducer_override, among other things, allows us to register a reducer that will be called for any class, independently of its type.

Notes:

reducer_override has the priority over dispatch_table-registered

reducers. * reducer_override can be used to fix other limitations of

cloudpickle for other types that suffered from type-specific reducers, such as Exceptions. See cloudpipe/cloudpickle#248

secretflow.utils.cloudpickle.code_position_independent_dumps(obj, protocol=None)[源代码]#

Serialize obj as a string of bytes allocated in memory.

Note that the serialization omits the code position info such as code filename, firstlineno, etc.

secretflow.utils.compressor#

Classes:

`Compressor`()	Abstract base class for cross device data compressor
`SparseCompressor`(sparse_rate)
`RandomSparse`(sparse_rate)	Random sparse compressor compress data by randomly set element to zero.
`TopkSparse`(sparse_rate)	Topk sparse compressor use topK algorithm to transfer dense matrix into sparse matrix.
`STCSparse`(sparse_rate)	Stc sparser, sample TopK element from original Weights TODO: rewrite in sl compress manner
`SCRSparse`(threshold)	Stc sparser, sample TopK element from original Weights TODO: rewrite in sl compress manner

Functions:

`sparse_encode`(data[, encode_method])	Encode the sparse matrix
`sparse_decode`(data)	Decode the compressed sparse matrix

class secretflow.utils.compressor.Compressor[源代码]#

基类：ABC

Abstract base class for cross device data compressor

Methods:

`compress`(data)	Compress data before send.
`decompress`(data)	Decompress data after receive.
`iscompressed`(data)	Checks whether data or data array has been compressed.

abstract compress(data: Union[ndarray, List[ndarray]]) → Union[Any, List[Any]][源代码]#

Compress data before send.

参数:: data (Union[np.ndarray, List[np.ndarray]]) – data need to compress.
返回:: compressed data.
返回类型:: Union[Any, List[Any]]

abstract decompress(data: Union[Any, List[Any]]) → Union[ndarray, List[ndarray]][源代码]#

Decompress data after receive.

参数:: data (Union[Any, List[Any]]) – data need to decompress.
返回:: decompressed data.
返回类型:: Union[np.ndarray, List[np.ndarray]]

abstract iscompressed(data: Union[Any, List[Any]]) → Union[bool, List[bool]][源代码]#

Checks whether data or data array has been compressed.

参数:: data (Union[Any, List[Any]]) – data need to check.
返回:: True if data is compressed.
返回类型:: Union[bool, List[bool]]

class secretflow.utils.compressor.SparseCompressor(sparse_rate: float)[源代码]#

基类：Compressor

Methods:

`__init__`(sparse_rate)	Initialize
`compress`(data)	Compress data to sparse matrix before send.
`decompress`(data)	Decompress data from sparse matrix to dense after received.
`iscompressed`(data)	Checks whether data or data array has been compressed.

__init__(sparse_rate: float)[源代码]#

Initialize

参数:: sparse_rate – the percentage of cells are zero.

compress(data: Union[ndarray, List[ndarray]]) → Union[spmatrix, List[spmatrix]][源代码]#

Compress data to sparse matrix before send.

参数:: data (Union[np.ndarray, List[np.ndarray]]) – data need to compress.
返回:: compressed data.
返回类型:: Union[sparse.spmatrix, List[sparse.spmatrix]]

decompress(data: Union[spmatrix, List[spmatrix]]) → Union[ndarray, List[ndarray]][源代码]#

Decompress data from sparse matrix to dense after received.

参数:: data (Union[sparse.spmatrix, List[sparse.spmatrix]]) – data need to decompress.
返回:: decompressed data.
返回类型:: Union[np.ndarray, List[np.ndarray]]

iscompressed(data: Union[spmatrix, List[spmatrix]]) → Union[bool, List[bool]][源代码]#

Checks whether data or data array has been compressed.

参数:: data (Union[sparse.spmatrix, List[sparse.spmatrix]]) – data need to check.
返回:: True if data is compressed.
返回类型:: Union[bool, List[bool]]

class secretflow.utils.compressor.RandomSparse(sparse_rate: float)[源代码]#

基类：SparseCompressor

Random sparse compressor compress data by randomly set element to zero.

Methods:

__init__(sparse_rate)

Initialize

__init__(sparse_rate: float)[源代码]#

Initialize

参数:: sparse_rate – the percentage of cells are zero.

class secretflow.utils.compressor.TopkSparse(sparse_rate: float)[源代码]#

基类：SparseCompressor

Topk sparse compressor use topK algorithm to transfer dense matrix into sparse matrix.

Methods:

__init__(sparse_rate)

Initialize

__init__(sparse_rate: float)[源代码]#

Initialize

参数:: sparse_rate – the percentage of cells are zero.

class secretflow.utils.compressor.STCSparse(sparse_rate: float)[源代码]#

基类：object

Stc sparser, sample TopK element from original Weights TODO: rewrite in sl compress manner

Methods:

__init__(sparse_rate)

__init__(sparse_rate: float)[源代码]#

class secretflow.utils.compressor.SCRSparse(threshold: float)[源代码]#

基类：object

Stc sparser, sample TopK element from original Weights TODO: rewrite in sl compress manner

Methods:

`__init__`(threshold)
`get_dimension`(index_value, threshold)

__init__(threshold: float)[源代码]#

get_dimension(index_value, threshold)[源代码]#

secretflow.utils.compressor.sparse_encode(data: List[ndarray], encode_method: str = 'coo') → List[源代码]#

Encode the sparse matrix

参数:

data – sparse matrix to be compressed
encode_method – compressed method,support [‘coo’, ‘gcxs’]

返回:

Compressed matrix

返回类型:

encoded_datas

secretflow.utils.compressor.sparse_decode(data: List) → List[ndarray][源代码]#

Decode the compressed sparse matrix

参数:: data – compressed matrix to be decoded
返回:: Decoded matrix
返回类型:: decoded_datas

secretflow.utils.errors#

Exceptions:

`AlreadyExistsError`	Raise when already exists.
`InvalidArgumentError`	Raise when invalid argument.
`NotFoundError`	Raise if not found.
`PartyNotFoundError`	Raise if party not found.
`UnexpectedError`	Raise when unexpected.
`HttpNotOkError`	Raise if http code is not 200

exception secretflow.utils.errors.AlreadyExistsError[源代码]#

基类：Exception

Raise when already exists.

exception secretflow.utils.errors.InvalidArgumentError[源代码]#

基类：Exception

Raise when invalid argument.

exception secretflow.utils.errors.NotFoundError[源代码]#

基类：Exception

Raise if not found.

exception secretflow.utils.errors.PartyNotFoundError[源代码]#

基类：Exception

Raise if party not found.

exception secretflow.utils.errors.UnexpectedError[源代码]#

基类：Exception

Raise when unexpected.

exception secretflow.utils.errors.HttpNotOkError[源代码]#

基类：Exception

Raise if http code is not 200

secretflow.utils.hash#

Functions:

sha256sum(filename)

secretflow.utils.hash.sha256sum(filename: str)[源代码]#

secretflow.utils.io#

Functions:

rows_count(filename)

get rows count from file

secretflow.utils.io.rows_count(filename)[源代码]#: get rows count from file

secretflow.utils.logging#

Functions:

`set_logging_level`(level)
`get_logging_level`()

secretflow.utils.logging.set_logging_level(level: str)[源代码]#

secretflow.utils.logging.get_logging_level()[源代码]#

secretflow.utils.ndarray_bigint#

Functions:

`randbits`(shape, bits)
`randint`(shape, min, max)
`arange`(max)
`zeros`(shape)

Classes:

BigintNdArray(data, shape)

secretflow.utils.ndarray_bigint.randbits(shape: tuple, bits)[源代码]#

secretflow.utils.ndarray_bigint.randint(shape: tuple, min, max)[源代码]#

secretflow.utils.ndarray_bigint.arange(max)[源代码]#

secretflow.utils.ndarray_bigint.zeros(shape)[源代码]#

class secretflow.utils.ndarray_bigint.BigintNdArray(data, shape)[源代码]#

基类：object

Methods:

`__init__`(data, shape)
`resize`(shape)
`to_list`()
`to_numpy`()
`to_hnp`(encoder)
`to_bytes`(bytes_per_int[, byteorder])

__init__(data, shape)[源代码]#

resize(shape)[源代码]#

to_list()[源代码]#

to_numpy()[源代码]#

to_hnp(encoder)[源代码]#

to_bytes(bytes_per_int, byteorder='little')[源代码]#

secretflow.utils.ndarray_encoding#

Functions:

`encode`(m, fxp_bits)	Encode float ndarray to uint64 finite field.
`decode`(m, fxp_bits)	Decode ndarray from uint64 finite field to the float.

secretflow.utils.ndarray_encoding.encode(m: ndarray, fxp_bits: int) → ndarray[源代码]#

Encode float ndarray to uint64 finite field. Float will times 2**fxp_bits firstly.

参数:

m (np.ndarray) – the ndarray to encode.
fraction_precision (int) – keep how many decimal digits after the dot. Must provide if ndarray dtype is float.

返回:

the encoded ndarray.

返回类型:

np.ndarray

secretflow.utils.ndarray_encoding.decode(m: ndarray, fxp_bits: int) → ndarray[源代码]#

Decode ndarray from uint64 finite field to the float. Fraction precision shall be corresponding to encoding fraction precision.

参数:

m (np.ndarray) – the ndarray to decode.
fxp_bits (int) – the decimal digits to keep when encoding float. Must provide if the original dtype is float.

返回:

the decoded float ndarray.

返回类型:

np.ndarray

secretflow.utils.random#

Functions:

global_random(device, exclusive_upper_bound)

secretflow.utils.random.global_random(device: PYU, exclusive_upper_bound: int) → int[源代码]#

secretflow.utils.ray_compatibility#

Functions:

ray_version_less_than_2_0_0()

Whther the current ray version is less 2.0.0.

secretflow.utils.ray_compatibility.ray_version_less_than_2_0_0()[源代码]#: Whther the current ray version is less 2.0.0.

secretflow.utils.sigmoid#

Functions:

`t1_sig`(x[, limit])	taylor series referenced from: https://mortendahl.github.io/2017/04/17/private-deep-learning-with-mpc/
`t3_sig`(x[, limit])	taylor series referenced from: https://mortendahl.github.io/2017/04/17/private-deep-learning-with-mpc/
`t5_sig`(x[, limit])	taylor series referenced from: https://mortendahl.github.io/2017/04/17/private-deep-learning-with-mpc/
`seg3_sig`(x)	f(x) = 0.5 + 0.125x if -4 <= x <= 4
`df_sig`(x)	https://dergipark.org.tr/en/download/article-file/54559 Dataflow implementation of sigmoid function: F(x) = 0.5 * ( x / ( 1 + \|x\| ) ) + 0.5 df_sig has higher precision than sr_sig if x in [-2, 2]
`sr_sig`(x)	https://en.wikipedia.org/wiki/Sigmoid_function#Examples Square Root approximation functions: F(x) = 0.5 * ( x / ( 1 + x^2 )^0.5 ) + 0.5 sr_sig almost perfect fit to sigmoid if x out of range [-3,3]
`ls7_sig`(x)	Polynomial fitting
`mix_sig`(x)	mix ls7 & sr sig, use ls7 if \|x\| < 4 , else use sr.
`real_sig`(x)
`sigmoid`(x, sig_type)

Classes:

SigType(value)

An enumeration.

secretflow.utils.sigmoid.t1_sig(x, limit: bool = True)[源代码]#: taylor series referenced from: https://mortendahl.github.io/2017/04/17/private-deep-learning-with-mpc/

secretflow.utils.sigmoid.t3_sig(x, limit: bool = True)[源代码]#: taylor series referenced from: https://mortendahl.github.io/2017/04/17/private-deep-learning-with-mpc/

secretflow.utils.sigmoid.t5_sig(x, limit: bool = True)[源代码]#: taylor series referenced from: https://mortendahl.github.io/2017/04/17/private-deep-learning-with-mpc/

secretflow.utils.sigmoid.seg3_sig(x)[源代码]#

f(x) = 0.5 + 0.125x if -4 <= x <= 4: 1 if x > 4 0 if -4 > x

secretflow.utils.sigmoid.df_sig(x)[源代码]#: https://dergipark.org.tr/en/download/article-file/54559 Dataflow implementation of sigmoid function: F(x) = 0.5 * ( x / ( 1 + |x| ) ) + 0.5 df_sig has higher precision than sr_sig if x in [-2, 2]

secretflow.utils.sigmoid.sr_sig(x)[源代码]#: https://en.wikipedia.org/wiki/Sigmoid_function#Examples Square Root approximation functions: F(x) = 0.5 * ( x / ( 1 + x^2 )^0.5 ) + 0.5 sr_sig almost perfect fit to sigmoid if x out of range [-3,3]

secretflow.utils.sigmoid.ls7_sig(x)[源代码]#: Polynomial fitting

secretflow.utils.sigmoid.mix_sig(x)[源代码]#: mix ls7 & sr sig, use ls7 if |x| < 4 , else use sr. has higher precision in all input range. NOTICE: this method is very expensive, only use for hessian matrix.

secretflow.utils.sigmoid.real_sig(x)[源代码]#

class secretflow.utils.sigmoid.SigType(value)[源代码]#

基类：Enum

An enumeration.

Attributes:

`REAL`
`T1`
`T3`
`T5`
`DF`
`SR`
`MIX`

REAL = 'real'#

T1 = 't1'#

T3 = 't3'#

T5 = 't5'#

DF = 'df'#

SR = 'sr'#

MIX = 'mix'#

secretflow.utils.sigmoid.sigmoid(x, sig_type: SigType)[源代码]#

secretflow.utils.testing#

Functions:

`unused_tcp_port`()	Return an unused port
`cluster_def`(parties[, runtime_config])	Generate SPU cluster_def for testing purposes.
`heu_config`(sk_keeper, evaluators)

secretflow.utils.testing.unused_tcp_port() → int[源代码]#: Return an unused port

secretflow.utils.testing.cluster_def(parties: List[str], runtime_config=None) → Dict[str, Any][源代码]#

Generate SPU cluster_def for testing purposes.

参数:

parties (List[str]) – parties of SPU devices. Must be more than 1 party,
runtime_config (optional) –
runtime_config of SPU device. More details refer to

SPU runtime config.
Defaults to None and use default runtime config.
1. If 3 parties are present, protocol would be set to ABY3 and field to FM128.
2. Otherwise, protocol would be set to SEMI2k and field to FM128. Other options are using default values.

返回:

An SPU cluster_def to initiate an SPU device.

返回类型:

Dict[str, Any]

secretflow.utils.testing.heu_config(sk_keeper: str, evaluators: List[str])[源代码]#