secretflow.utils#

secretflow.utils.simulation

secretflow.utils.cloudpickle#

This file mainly borrows from ray.cloudpickle.cloudpickle_fast.py.

We introduce a custom serialization omitting code positions, e.g., function filename, first line no, .etc.

It is useful for some scenarios that need to generate the signature of a function deterministicly and is used to sign python functions to execute in TEE.

Classes:

CodePositionIndependentCloudPickler(file[, ...])

Functions:

code_position_independent_dumps(obj[, protocol])

Serialize obj as a string of bytes allocated in memory.

class secretflow.utils.cloudpickle.CodePositionIndependentCloudPickler(file, protocol=None, buffer_callback=None)[source]#

Bases: CloudPickler

Attributes:

dispatch_table

Methods:

`dump`(obj)	Write a pickled representation of the given object to the open file.
`__init__`(file[, protocol, buffer_callback])
`reducer_override`(obj)	Type-agnostic reducing callback for function and classes.

dispatch_table = ChainMap({<class 'classmethod'>: <function _classmethod_reduce>, <class '_io.TextIOWrapper'>: <function _file_reduce>, <class 'logging.Logger'>: <function _logger_reduce>, <class 'logging.RootLogger'>: <function _root_logger_reduce>, <class 'memoryview'>: <function _memoryview_reduce>, <class 'property'>: <function _property_reduce>, <class 'staticmethod'>: <function _classmethod_reduce>, <class 'cell'>: <function _cell_reduce>, <class 'code'>: <function _code_reduce>, <class 'getset_descriptor'>: <function _getset_descriptor_reduce>, <class 'module'>: <function _module_reduce>, <class 'method'>: <function _method_reduce>, <class 'mappingproxy'>: <function _mappingproxy_reduce>, <class '_weakrefset.WeakSet'>: <function _weakset_reduce>, <class 'typing.TypeVar'>: <function _typevar_reduce>, <class 'dict_keys'>: <function _dict_keys_reduce>, <class 'dict_values'>: <function _dict_values_reduce>, <class 'dict_items'>: <function _dict_items_reduce>, <class 'odict_keys'>: <function _odict_keys_reduce>, <class 'odict_values'>: <function _odict_values_reduce>, <class 'odict_items'>: <function _odict_items_reduce>}, {<class 'complex'>: <function pickle_complex>, <class 're.Pattern'>: <function _pickle>, <class 'soupsieve.css_types.Selector'>: <function _pickle>, <class 'soupsieve.css_types.SelectorNull'>: <function _pickle>, <class 'soupsieve.css_types.SelectorTag'>: <function _pickle>, <class 'soupsieve.css_types.SelectorAttribute'>: <function _pickle>, <class 'soupsieve.css_types.SelectorContains'>: <function _pickle>, <class 'soupsieve.css_types.SelectorNth'>: <function _pickle>, <class 'soupsieve.css_types.SelectorLang'>: <function _pickle>, <class 'soupsieve.css_types.SelectorList'>: <function _pickle>, <class 'soupsieve.css_match.SoupSieve'>: <function _pickle>, <class 'numpy.ufunc'>: <function _ufunc_reduce>, <class 'numpy._DTypeMeta'>: <function _DType_reduce>, <class 'uarray._Function'>: <function pickle_function>, <class 'uarray._BackendState'>: <function pickle_state>, <class 'uarray._SetBackendContext'>: <function pickle_set_backend_context>, <class 'uarray._SkipBackendContext'>: <function pickle_skip_backend_context>, <class 'torch.layout'>: <function <lambda>>})#

dump(obj)[source]#: Write a pickled representation of the given object to the open file.

__init__(file, protocol=None, buffer_callback=None)[source]#

reducer_override(obj)[source]#

Type-agnostic reducing callback for function and classes.

For performance reasons, subclasses of the C _pickle.Pickler class cannot register custom reducers for functions and classes in the dispatch_table. Reducer for such types must instead implemented in the special reducer_override method.

Note that method will be called for any object except a few builtin-types (int, lists, dicts etc.), which differs from reducers in the Pickler’s dispatch_table, each of them being invoked for objects of a specific type only.

This property comes in handy for classes: although most classes are instances of the type metaclass, some of them can be instances of other custom metaclasses (such as enum.EnumMeta for example). In particular, the metaclass will likely not be known in advance, and thus cannot be special-cased using an entry in the dispatch_table. reducer_override, among other things, allows us to register a reducer that will be called for any class, independently of its type.

Notes:

reducer_override has the priority over dispatch_table-registered

reducers. * reducer_override can be used to fix other limitations of

cloudpickle for other types that suffered from type-specific reducers, such as Exceptions. See cloudpipe/cloudpickle#248

secretflow.utils.cloudpickle.code_position_independent_dumps(obj, protocol=None)[source]#

Serialize obj as a string of bytes allocated in memory.

Note that the serialization omits the code position info such as code filename, firstlineno, etc.

secretflow.utils.compressor#

Classes:

`Compressor`()	Abstract base class for cross device data compressor
`SparseCompressor`(sparse_rate)
`RandomSparse`(sparse_rate)	Random sparse compressor compress data by randomly set element to zero.
`TopkSparse`(sparse_rate)	Topk sparse compressor use topK algorithm to transfer dense matrix into sparse matrix.
`STCSparse`(sparse_rate)	Stc sparser, sample TopK element from original Weights TODO: rewrite in sl compress manner
`SCRSparse`(threshold)	Stc sparser, sample TopK element from original Weights TODO: rewrite in sl compress manner

Functions:

`sparse_encode`(data[, encode_method])	Encode the sparse matrix
`sparse_decode`(data)	Decode the compressed sparse matrix

class secretflow.utils.compressor.Compressor[source]#

Bases: ABC

Abstract base class for cross device data compressor

Methods:

`compress`(data)	Compress data before send.
`decompress`(data)	Decompress data after receive.
`iscompressed`(data)	Checks whether data or data array has been compressed.

abstract compress(data: Union[ndarray, List[ndarray]]) → Union[Any, List[Any]][source]#

Compress data before send.

Parameters:: data (Union[np.ndarray, List[np.ndarray]]) – data need to compress.
Returns:: compressed data.
Return type:: Union[Any, List[Any]]

abstract decompress(data: Union[Any, List[Any]]) → Union[ndarray, List[ndarray]][source]#

Decompress data after receive.

Parameters:: data (Union[Any, List[Any]]) – data need to decompress.
Returns:: decompressed data.
Return type:: Union[np.ndarray, List[np.ndarray]]

abstract iscompressed(data: Union[Any, List[Any]]) → Union[bool, List[bool]][source]#

Checks whether data or data array has been compressed.

Parameters:: data (Union[Any, List[Any]]) – data need to check.
Returns:: True if data is compressed.
Return type:: Union[bool, List[bool]]

class secretflow.utils.compressor.SparseCompressor(sparse_rate: float)[source]#

Bases: Compressor

Methods:

`__init__`(sparse_rate)	Initialize
`compress`(data)	Compress data to sparse matrix before send.
`decompress`(data)	Decompress data from sparse matrix to dense after received.
`iscompressed`(data)	Checks whether data or data array has been compressed.

__init__(sparse_rate: float)[source]#

Initialize

Parameters:: sparse_rate – the percentage of cells are zero.

compress(data: Union[ndarray, List[ndarray]]) → Union[spmatrix, List[spmatrix]][source]#

Compress data to sparse matrix before send.

Parameters:: data (Union[np.ndarray, List[np.ndarray]]) – data need to compress.
Returns:: compressed data.
Return type:: Union[sparse.spmatrix, List[sparse.spmatrix]]

decompress(data: Union[spmatrix, List[spmatrix]]) → Union[ndarray, List[ndarray]][source]#

Decompress data from sparse matrix to dense after received.

Parameters:: data (Union[sparse.spmatrix, List[sparse.spmatrix]]) – data need to decompress.
Returns:: decompressed data.
Return type:: Union[np.ndarray, List[np.ndarray]]

iscompressed(data: Union[spmatrix, List[spmatrix]]) → Union[bool, List[bool]][source]#

Checks whether data or data array has been compressed.

Parameters:: data (Union[sparse.spmatrix, List[sparse.spmatrix]]) – data need to check.
Returns:: True if data is compressed.
Return type:: Union[bool, List[bool]]

class secretflow.utils.compressor.RandomSparse(sparse_rate: float)[source]#

Bases: SparseCompressor

Random sparse compressor compress data by randomly set element to zero.

Methods:

__init__(sparse_rate)

Initialize

__init__(sparse_rate: float)[source]#

Initialize

Parameters:: sparse_rate – the percentage of cells are zero.

class secretflow.utils.compressor.TopkSparse(sparse_rate: float)[source]#

Bases: SparseCompressor

Topk sparse compressor use topK algorithm to transfer dense matrix into sparse matrix.

Methods:

__init__(sparse_rate)

Initialize

__init__(sparse_rate: float)[source]#

Initialize

Parameters:: sparse_rate – the percentage of cells are zero.

class secretflow.utils.compressor.STCSparse(sparse_rate: float)[source]#

Bases: object

Stc sparser, sample TopK element from original Weights TODO: rewrite in sl compress manner

Methods:

__init__(sparse_rate)

__init__(sparse_rate: float)[source]#

class secretflow.utils.compressor.SCRSparse(threshold: float)[source]#

Bases: object

Stc sparser, sample TopK element from original Weights TODO: rewrite in sl compress manner

Methods:

`__init__`(threshold)
`get_dimension`(index_value, threshold)

__init__(threshold: float)[source]#

get_dimension(index_value, threshold)[source]#

secretflow.utils.compressor.sparse_encode(data: List[ndarray], encode_method: str = 'coo') → List[source]#

Encode the sparse matrix

Parameters:

data – sparse matrix to be compressed
encode_method – compressed method,support [‘coo’, ‘gcxs’]

Returns:

Compressed matrix

Return type:

encoded_datas

secretflow.utils.compressor.sparse_decode(data: List) → List[ndarray][source]#

Decode the compressed sparse matrix

Parameters:: data – compressed matrix to be decoded
Returns:: Decoded matrix
Return type:: decoded_datas

secretflow.utils.errors#

Exceptions:

`AlreadyExistsError`	Raise when already exists.
`InvalidArgumentError`	Raise when invalid argument.
`NotFoundError`	Raise if not found.
`PartyNotFoundError`	Raise if party not found.
`UnexpectedError`	Raise when unexpected.
`HttpNotOkError`	Raise if http code is not 200

exception secretflow.utils.errors.AlreadyExistsError[source]#

Bases: Exception

Raise when already exists.

exception secretflow.utils.errors.InvalidArgumentError[source]#

Bases: Exception

Raise when invalid argument.

exception secretflow.utils.errors.NotFoundError[source]#

Bases: Exception

Raise if not found.

exception secretflow.utils.errors.PartyNotFoundError[source]#

Bases: Exception

Raise if party not found.

exception secretflow.utils.errors.UnexpectedError[source]#

Bases: Exception

Raise when unexpected.

exception secretflow.utils.errors.HttpNotOkError[source]#

Bases: Exception

Raise if http code is not 200

secretflow.utils.hash#

Functions:

sha256sum(filename)

secretflow.utils.hash.sha256sum(filename: str)[source]#

secretflow.utils.io#

Functions:

rows_count(filename)

get rows count from file

secretflow.utils.io.rows_count(filename)[source]#: get rows count from file

secretflow.utils.logging#

Functions:

`set_logging_level`(level)
`get_logging_level`()

secretflow.utils.logging.set_logging_level(level: str)[source]#

secretflow.utils.logging.get_logging_level()[source]#

secretflow.utils.ndarray_bigint#

Functions:

`randbits`(shape, bits)
`randint`(shape, min, max)
`arange`(max)
`zeros`(shape)

Classes:

BigintNdArray(data, shape)

secretflow.utils.ndarray_bigint.randbits(shape: tuple, bits)[source]#

secretflow.utils.ndarray_bigint.randint(shape: tuple, min, max)[source]#

secretflow.utils.ndarray_bigint.arange(max)[source]#

secretflow.utils.ndarray_bigint.zeros(shape)[source]#

class secretflow.utils.ndarray_bigint.BigintNdArray(data, shape)[source]#

Bases: object

Methods:

`__init__`(data, shape)
`resize`(shape)
`to_list`()
`to_numpy`()
`to_hnp`(encoder)
`to_bytes`(bytes_per_int[, byteorder])

__init__(data, shape)[source]#

resize(shape)[source]#

to_list()[source]#

to_numpy()[source]#

to_hnp(encoder)[source]#

to_bytes(bytes_per_int, byteorder='little')[source]#

secretflow.utils.ndarray_encoding#

Functions:

`encode`(m, fxp_bits)	Encode float ndarray to uint64 finite field.
`decode`(m, fxp_bits)	Decode ndarray from uint64 finite field to the float.

secretflow.utils.ndarray_encoding.encode(m: ndarray, fxp_bits: int) → ndarray[source]#

Encode float ndarray to uint64 finite field. Float will times 2**fxp_bits firstly.

Parameters:

m (np.ndarray) – the ndarray to encode.
fraction_precision (int) – keep how many decimal digits after the dot. Must provide if ndarray dtype is float.

Returns:

the encoded ndarray.

Return type:

np.ndarray

secretflow.utils.ndarray_encoding.decode(m: ndarray, fxp_bits: int) → ndarray[source]#

Decode ndarray from uint64 finite field to the float. Fraction precision shall be corresponding to encoding fraction precision.

Parameters:

m (np.ndarray) – the ndarray to decode.
fxp_bits (int) – the decimal digits to keep when encoding float. Must provide if the original dtype is float.

Returns:

the decoded float ndarray.

Return type:

np.ndarray

secretflow.utils.random#

Functions:

global_random(device, exclusive_upper_bound)

secretflow.utils.random.global_random(device: PYU, exclusive_upper_bound: int) → int[source]#

secretflow.utils.ray_compatibility#

Functions:

ray_version_less_than_2_0_0()

Whther the current ray version is less 2.0.0.

secretflow.utils.ray_compatibility.ray_version_less_than_2_0_0()[source]#: Whther the current ray version is less 2.0.0.

secretflow.utils.sigmoid#

Functions:

`t1_sig`(x[, limit])	taylor series referenced from: https://mortendahl.github.io/2017/04/17/private-deep-learning-with-mpc/
`t3_sig`(x[, limit])	taylor series referenced from: https://mortendahl.github.io/2017/04/17/private-deep-learning-with-mpc/
`t5_sig`(x[, limit])	taylor series referenced from: https://mortendahl.github.io/2017/04/17/private-deep-learning-with-mpc/
`seg3_sig`(x)	f(x) = 0.5 + 0.125x if -4 <= x <= 4
`df_sig`(x)	https://dergipark.org.tr/en/download/article-file/54559 Dataflow implementation of sigmoid function: F(x) = 0.5 * ( x / ( 1 + \|x\| ) ) + 0.5 df_sig has higher precision than sr_sig if x in [-2, 2]
`sr_sig`(x)	https://en.wikipedia.org/wiki/Sigmoid_function#Examples Square Root approximation functions: F(x) = 0.5 * ( x / ( 1 + x^2 )^0.5 ) + 0.5 sr_sig almost perfect fit to sigmoid if x out of range [-3,3]
`ls7_sig`(x)	Polynomial fitting
`mix_sig`(x)	mix ls7 & sr sig, use ls7 if \|x\| < 4 , else use sr.
`real_sig`(x)
`sigmoid`(x, sig_type)

Classes:

SigType(value)

An enumeration.

secretflow.utils.sigmoid.t1_sig(x, limit: bool = True)[source]#: taylor series referenced from: https://mortendahl.github.io/2017/04/17/private-deep-learning-with-mpc/

secretflow.utils.sigmoid.t3_sig(x, limit: bool = True)[source]#: taylor series referenced from: https://mortendahl.github.io/2017/04/17/private-deep-learning-with-mpc/

secretflow.utils.sigmoid.t5_sig(x, limit: bool = True)[source]#: taylor series referenced from: https://mortendahl.github.io/2017/04/17/private-deep-learning-with-mpc/

secretflow.utils.sigmoid.seg3_sig(x)[source]#

f(x) = 0.5 + 0.125x if -4 <= x <= 4: 1 if x > 4 0 if -4 > x

secretflow.utils.sigmoid.df_sig(x)[source]#: https://dergipark.org.tr/en/download/article-file/54559 Dataflow implementation of sigmoid function: F(x) = 0.5 * ( x / ( 1 + |x| ) ) + 0.5 df_sig has higher precision than sr_sig if x in [-2, 2]

secretflow.utils.sigmoid.sr_sig(x)[source]#: https://en.wikipedia.org/wiki/Sigmoid_function#Examples Square Root approximation functions: F(x) = 0.5 * ( x / ( 1 + x^2 )^0.5 ) + 0.5 sr_sig almost perfect fit to sigmoid if x out of range [-3,3]

secretflow.utils.sigmoid.ls7_sig(x)[source]#: Polynomial fitting

secretflow.utils.sigmoid.mix_sig(x)[source]#: mix ls7 & sr sig, use ls7 if |x| < 4 , else use sr. has higher precision in all input range. NOTICE: this method is very expensive, only use for hessian matrix.

secretflow.utils.sigmoid.real_sig(x)[source]#

class secretflow.utils.sigmoid.SigType(value)[source]#

Bases: Enum

An enumeration.

Attributes:

`REAL`
`T1`
`T3`
`T5`
`DF`
`SR`
`MIX`

REAL = 'real'#

T1 = 't1'#

T3 = 't3'#

T5 = 't5'#

DF = 'df'#

SR = 'sr'#

MIX = 'mix'#

secretflow.utils.sigmoid.sigmoid(x, sig_type: SigType)[source]#

secretflow.utils.testing#

Functions:

`unused_tcp_port`()	Return an unused port
`cluster_def`(parties[, runtime_config])	Generate SPU cluster_def for testing purposes.
`heu_config`(sk_keeper, evaluators)

secretflow.utils.testing.unused_tcp_port() → int[source]#: Return an unused port

secretflow.utils.testing.cluster_def(parties: List[str], runtime_config=None) → Dict[str, Any][source]#

Generate SPU cluster_def for testing purposes.

Parameters:

parties (List[str]) – parties of SPU devices. Must be more than 1 party,
runtime_config (optional) –
runtime_config of SPU device. More details refer to

SPU runtime config.
Defaults to None and use default runtime config.
1. If 3 parties are present, protocol would be set to ABY3 and field to FM128.
2. Otherwise, protocol would be set to SEMI2k and field to FM128. Other options are using default values.

Returns:

An SPU cluster_def to initiate an SPU device.

Return type:

Dict[str, Any]

secretflow.utils.testing.heu_config(sk_keeper: str, evaluators: List[str])[source]#