Deployment#

Pre-knowledge: about Ray#

SecretFlow uses Ray as its distributed framework. A Ray cluster consists of a head node and zero or several slave nodes, for more information about Ray, please visit Ray official website.

Simulation#

SecretFlow is designed for fast simulation on a single host or on multiple nodes with single ray cluster.

Note

SecretFlow with single ray cluster is for simulation only. Please refer to production section below for production.

Standalone mode for simulation#

Use secretflow.init directly to run secretflow in standalone mode. A ray cluster with only one node will be started, and it will automatically shut down when the program exits.

>>> import secretflow as sf
>>> sf.init(parties=['alice', 'bob'], address='local')

Cluster mode for simulation#

In the cluster simulation mode, each Ray node simulates an participant. This is done by adding the participant name as resource tag on each Ray node, and then the calculation task of each participant is dispatched to the corresponding Ray node. The overall communication network is as follows.

simulation_comm

The following is an example showing how to build a cluster consisting of alice and bob on multiple nodes.

Start head node#

Start a head node on your first machine with the tag “alice”.

NOTE

  1. Remember to use the real ip and port instead.

  2. {"alice": 16} means that alice can run up to 16 workers at the same time. Just feel free to change it if you like.

ray start --head --node-ip-address="ip" --port="port" --resources='{"alice": 16}' --include-dashboard=False --disable-usage-stats

Head node starts successfully if you see “Ray runtime started.” in the screen output.

Now we have a cluster with a head node only, let us start more nodes.

Start other nodes#

Start a node with the tag “bob” on another machine. The node will connect to the head node and join the cluster.

Note

Replace ip:port with the node-ip-address and port of head node please.

ray start --address="ip:port" --resources='{"bob": 16}' --disable-usage-stats

The node starts successfully if you see “Ray runtime started.” in the screen output. A Ray cluster consisting of two Ray nodes has been built by now, while the head node simulates alice and the slave node simulates bob.

You can repeat the step above to start more nodes with using other parties as resources tag as you like.

Start SecretFlow#

Now you can start SecretFlow and run your code. The following code shows that alice and bob each execute a function that returns the input value.

Tips

  1. Replace ip:port in sf.init with the node-ip-address and port of head node please.

  2. If you start more nodes (such as carol, davy, etc.), remember to add the new party name in the parties=['alice', 'bob'] parameter.

>>> import secretflow as sf
# Replace with the `node-ip-address` and `port` of head node.
>>> sf.init(parties=['alice', 'bob'], address='ip:port')
>>> alice = sf.PYU('alice')
>>> bob = sf.PYU('bob')
>>> alice(lambda x : x)(2)
<secretflow.device.device.pyu.PYUObject object at 0x7fe932a1a640>
>>> bob(lambda x : x)(2)
<secretflow.device.device.pyu.PYUObject object at 0x7fe6fef03250>

(optional) How to shut down the cluster#

In some cases you would like to shut down the cluster, the following command will help you. Remember to run the command on all machines.

Note that all ray processors on the machine will be stopped, which means all ray clusters will be stopped.

ray stop

(optional) How to setup an SPU in cluster mode#

SPU consists of multi workers on different nodes. For performance reasons, the major part of SPU is written in C++. SPU is based on Brpc, which indicates it has a separated service mesh independent of Ray’s networking. In a word, you need to assign different ports for the SPU for now. We are working on merging them.

A typical SPU config is as follows.

Tips

  1. Replace address in sf.init with the node-ip-address and port of head node please.

  2. Fill address of alice with the address which can be accessed by bob and choose an unused port different with Ray.

  3. listen_addr of alice can use the same port of alice address.

  4. Fill address of bob with the ip which can be accessed by alice and choose an unused port different with Ray.

  5. listen_addr of bob can use the same port of bob address.

import spu
import secretflow as sf

# Use ray head adress please.
sf.init(parties=['alice', 'bob'], address='Ray head node address')

cluster_def={
    'nodes': [
        {
            'party': 'alice',
            # Please choose an unused port.
            'address': 'ip:port of alice',
            'listen_addr': '0.0.0.0:port'
        },
        {
            'party': 'bob',
            # Please choose an unused port.
            'address': 'ip:port of bob',
            'listen_addr': '0.0.0.0:port'
        },
    ],
    'runtime_config': {
        'protocol': spu.spu_pb2.SEMI2K,
        'field': spu.spu_pb2.FM128,
        'sigmoid_mode': spu.spu_pb2.RuntimeConfig.SIGMOID_REAL,
    }
}

spu = sf.SPU(cluster_def=cluster_def)

For more configurations of SPU, please refer to SPU config

Note

You will see the usage of setup an spu in many tutorials. But be careful that it works only in standalone mode because sf.utils.testing.cluster_def use 127.0.0.1 as the default ip.

>>> spu = sf.SPU(sf.utils.testing.cluster_def(['alice', 'bob', 'carol']))

Deploy SecretFlow in a docker container with simulation mode#

You may need to understand the concept of docker network firstly. The two main docker networks are host network and bridge network, you can click the link to read the official documentation to learn more.”

SecretFlow recommends to start the container using the host network mode, and the following will explain the reasons to you.

Production#

SecretFlow provides multi controller mode for production with enhanced security. (If you want to know more, welcome to read Programming in SecretFlow)

A SecretFlow cluster for production consists of several ray clusters, and every party has its own ray cluster. Each participant must execute the code at the same time to complete the task of collaboration. The architecture of the production mode is shown in the figure below.

simulation_comm

The following will guide you to deploy SecretFlow for production.

Setup a SecretFlow cluster crossing silo#

The following is an example showing how to build a cluster consisting of alice and bob for production.

Note

Please keep in mind that alice and bob should run the code simultaneously.

Start SecretFlow on the node of alice#

alice starts its ray cluster firstly. Note that the command here is to start Ray’s head node.

ray start --head --node-ip-address="ip" --port="port" --include-dashboard=False --disable-usage-stats

Head node starts successfully if you see “Ray runtime started.” in the screen output. So far, Alice’s Ray cluster has been successfully built.

Then alice initializes SecretFlow with a cluster config and runs the code.

Tips

  1. Replace ip:port in sf.init with the node-ip-address and port of head node please.

  2. Fill address of alice with the address which can be accessed by bob. Remember to choose an unused port different with port of Ray and SPU.

  3. Fill address of bob with the address which can be accessed by alice. Remember to choose an unused port different with port of Ray and SPU.

  4. Note that self_party is alice.

  5. Please note that sf.init does not need to provide the parties parameter, but needs to provide a cluster_config to describe the communication address and port between the two organizations.

cluster_config ={
    'parties': {
        'alice': {
            # replace with alice's real address.
            'address': 'ip:port of alice',
            'listen_addr': '0.0.0.0:port'
        },
        'bob': {
            # replace with bob's real address.
            'address': 'ip:port of bob',
            'listen_addr': '0.0.0.0:port'
        },
    },
    'self_party': 'alice'
}

sf.init(address='alice ray head node address', cluster_config=cluster_config)

# your code to run.

Start SecretFlow on the node of bob#

bob starts its ray cluster firstly. Note that the command here is to start the head node of Ray, because bob needs to build a separate Ray cluster.

ray start --head --node-ip-address="ip" --port="port" --include-dashboard=False --disable-usage-stats

Head node starts successfully if you see “Ray runtime started.” in the screen output. So far, Alice’s Ray cluster has been successfully built.

Then bob initializes SecretFlow with a cluster config almost same as alice except for self_party and ray address and runs the code.

Tips

  1. Replace address in sf.init with the node-ip-address and port of head node please. Note, here is bob’s head node address, please don’t fill in alice’s.

  2. Fill address of alice with the address which can be accessed by bob. Remember to choose an unused port different with port of Ray and SPU.

  3. Fill address of bob with the address which can be accessed by alice. Remember to choose an unused port different with port of Ray and SPU.

  4. Note that self_party is bob.

  5. Please note that sf.init does not need to provide the parties parameter, but needs to provide a cluster_config to describe the communication address and port between the two organizations.


cluster_config ={
    'parties': {
        'alice': {
            # replace with alice's real address.
            'address': 'ip:port of alice',
            'listen_addr': '0.0.0.0:port'
        },
        'bob': {
            # replace with bob's real address.
            'address': 'ip:port of bob',
            'listen_addr': '0.0.0.0:port'
        },
    },
    'self_party': 'bob'
}

sf.init(address='bob ray head node address', cluster_config=cluster_config)

# your code to run.

How to setup SPU for production#

The way to build the SPU is the same as the simulation mode, the only difference is that building the SPU also needs to be executed by multiple parties at the same time, please read the previous article for more details.

In order to avoid problems such as connection timeout caused by the startup time difference, you may need to set the link_desc parameter when initialize the SPU to adjust the connection-related parameters. For details, see SPU .

Suggestions for production#

  1. Enable tls Authentication.

    SecretFlow can be configured to use TLS on cross-silo gRPC channels.

    An example for alice.

    tls_config = {
    "ca_cert": "ca root cert of other parties (e.g. bob)",
    "cert": "server cert of alice in pem",
    "key": "server key of alice in pem",
}

sf.init(address='ip:port', 
        cluster_config=cluster_config, 
        tls_config=tls_config
)

    An example for bob.

    tls_config = {
    "ca_cert": "ca root cert of other parties (e.g. alice)",
    "cert": "server cert of bob in pem",
    "key": "server key of bob in pem",
}

sf.init(address='ip:port', 
        cluster_config=cluster_config, 
        tls_config=tls_config
)

  2. Enhanced serialization/deserialization.

    SecretFlow uses pickle in serialization/deserialization which is vulnerable. You can set cross_silo_serializing_allowed_list when init SecretFlow to specify an allowlist to restrict serializable objects. An example could be （You should not use this demo directly. Configure it to your actual needs.）

    allowed_list =  {
    "numpy.core.numeric": ["*"],
    "numpy": ["dtype"],
}

sf.init(address='ip:port', 
        cluster_config=cluster_config, 
        cross_silo_serializing_allowed_list=allowed_list
)