Horizontally Federated XGBoost#

The following codes are demos only. It’s NOT for production due to system security concerns, please DO NOT use it directly in production.

In this tutorial, we will learn how to use SecretFlow to train tree models for horizontal federation. Secretflow provides tree modeling capabilities for horizontal scenarios(SFXgboost), The usage of SFXgboost is similar to XGBoost, you can easily convert your existing XGBoost program into a federated model for SecretFlow.

Xgboost#

XGBoost is an optimized distributed gradient boosting library designed to be highly efficient, flexible and portable. It implements machine learning algorithms under the Gradient Boosting framework

official tutorial XGBoost tutorials

prepare secretflow devices#

[1]:

%load_ext autoreload
%autoreload 2

import secretflow as sf
# In case you have a running secretflow runtime already.
sf.shutdown()

sf.init(['alice', 'bob', 'charlie'], address='local')
alice, bob, charlie = sf.PYU('alice'), sf.PYU('bob'), sf.PYU('charlie')

2022-08-19 13:47:10.795519: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/rh/rh-ruby25/root/usr/local/lib64:/opt/rh/rh-ruby25/root/usr/lib64:/opt/rh/devtoolset-11/root/usr/lib64:/opt/rh/devtoolset-11/root/usr/lib:/opt/rh/devtoolset-11/root/usr/lib64/dyninst:/opt/rh/devtoolset-11/root/usr/lib/dyninst

XGBoost Example#

[2]:

import xgboost as xgb
import pandas as pd
from secretflow.utils.simulation.datasets import dataset

df = pd.read_csv(dataset('dermatology'))
df.fillna(value=0)
print(df.dtypes)
y = df['class']
y = y - 1
x = df.drop(columns="class")
dtrain = xgb.DMatrix(x, y)
dtest = dtrain
params = {
    'max_depth': 4,
    'objective': 'multi:softmax',
    'min_child_weight': 1,
    'max_bin': 10,
    'num_class': 6,
    'eval_metric': 'merror',
}
num_round = 4
watchlist = [(dtrain, 'train')]
bst = xgb.train(params, dtrain, num_round, evals=watchlist, early_stopping_rounds=2)

/home/fengjun.feng/miniconda3/envs/py3.8/lib/python3.8/site-packages/xgboost/compat.py:36: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  from pandas import MultiIndex, Int64Index
/home/fengjun.feng/miniconda3/envs/py3.8/lib/python3.8/site-packages/xgboost/data.py:262: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  elif isinstance(data.columns, (pd.Int64Index, pd.RangeIndex)):

erythema                                      int64
scaling                                       int64
definite_borders                              int64
itching                                       int64
koebner_phenomenon                            int64
polygonal_papules                             int64
follicular_papules                            int64
oral_mucosal_involvement                      int64
knee_and_elbow_involvement                    int64
scalp_involvement                             int64
family_history                                int64
melanin_incontinence                          int64
eosinophils_in_the_infiltrate                 int64
pnl_infiltrate                                int64
fibrosis_of_the_papillary_dermis              int64
exocytosis                                    int64
acanthosis                                    int64
hyperkeratosis                                int64
parakeratosis                                 int64
clubbing_of_the_rete_ridges                   int64
elongation_of_the_rete_ridges                 int64
thinning_of_the_suprapapillary_epidermis      int64
spongiform_pustule                            int64
munro_microabcess                             int64
focal_hypergranulosis                         int64
disappearance_of_the_granular_layer           int64
vacuolisation_and_damage_of_basal_layer       int64
spongiosis                                    int64
saw-tooth_appearance_of_retes                 int64
follicular_horn_plug                          int64
perifollicular_parakeratosis                  int64
inflammatory_monoluclear_inflitrate           int64
band-like_infiltrate                          int64
age                                         float64
class                                         int64
dtype: object
[0]     train-merror:0.01913
[1]     train-merror:0.01366
[2]     train-merror:0.01366
[3]     train-merror:0.00820

Then, How to do federated xgboost in secretflow?#

  1. Use federate Binning method based on iteration to calculate the global bucket information combined with the data of all sides, which was used as the candidate to enter the subsequent construction procedure

  2. The data is input into each Client XGBoost engine to calculate G & H

  3. Train federated boosting model

    1. Data is reassigned to the node to be split

    2. The sum of grad and the sum of hess are calculated according to the previously calculated binning buckets

    3. Send the sum of grad and the sum of hess to server，server use secure aggregation to produce global summary，then choose best split point，Send best split info back to clients.

    4. Clients Updates local model

  4. Finish training，and save model

Create 3 entities in the Secretflow environment [Alice, Bob, Charlie] Where ‘Alice’ and ‘Bob’ are clients, and Charlie is the server,then you can happily start Federate Boosting.

Prepare Data#

[3]:

from secretflow.data.horizontal import read_csv
from secretflow.security.aggregation import SecureAggregator
from secretflow.security.compare import SPUComparator
from secretflow.utils.simulation.datasets import load_dermatology

aggr = SecureAggregator(charlie, [alice, bob])
spu = sf.SPU(sf.utils.testing.cluster_def(['alice', 'bob']))
comp = SPUComparator(spu)
data = load_dermatology(parts=[alice, bob], aggregator=aggr, comparator=comp)
data.fillna(value=0, inplace=True)

(pid=3817970) 2022-08-19 13:47:17.904107: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/rh/rh-ruby25/root/usr/local/lib64:/opt/rh/rh-ruby25/root/usr/lib64:/opt/rh/devtoolset-11/root/usr/lib64:/opt/rh/devtoolset-11/root/usr/lib:/opt/rh/devtoolset-11/root/usr/lib64/dyninst:/opt/rh/devtoolset-11/root/usr/lib/dyninst
(pid=3817969) 2022-08-19 13:47:17.904107: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/rh/rh-ruby25/root/usr/local/lib64:/opt/rh/rh-ruby25/root/usr/lib64:/opt/rh/devtoolset-11/root/usr/lib64:/opt/rh/devtoolset-11/root/usr/lib:/opt/rh/devtoolset-11/root/usr/lib64/dyninst:/opt/rh/devtoolset-11/root/usr/lib/dyninst

Prepare Params#

[4]:

params = {
    # XGBoost parameter tutorial
    # https://xgboost.readthedocs.io/en/latest/parameter.html
    'max_depth': 4,  # max depth
    'eta': 0.3,  # learning rate
    'objective': 'multi:softmax',  # objection function，support "binary:logistic","reg:logistic","multi:softmax","multi:softprob","reg:squarederror"
    'min_child_weight': 1,  # The minimum value of weight
    'lambda': 0.1,  # L2 regularization term on weights (xgb's lambda)
    'alpha': 0,  # L1 regularization term on weights (xgb's alpha)
    'max_bin': 10,  # Max num of binning
    'num_class': 6,  # Only required in multi-class classification
    'gamma': 0,  # Same to min_impurity_split,The minimux gain for a split
    'subsample': 1.0,  # Subsample rate by rows
    'colsample_bytree': 1.0,  # Feature selection rate by tree
    'colsample_bylevel': 1.0,  # Feature selection rate by level
    'eval_metric': 'merror',  # supported eval metric：
    # 1. rmse
    # 2. rmsle
    # 3. mape
    # 4. logloss
    # 5. error
    # 6. error@t
    # 7. merror
    # 8. mlogloss
    # 9. auc
    # 10. aucpr
    # Special params in SFXgboost
    # Required
    'hess_key': 'hess',  # Required, Mark hess columns, optionally choosing a column name that is not in the data set
    'grad_key': 'grad',  # Required，Mark grad columns, optionally choosing a column name that is not in the data set
    'label_key': 'class',  # Required，ark label columns, optionally choosing a column name that is not in the data set
}

Create SFXgboost#

[6]:

from secretflow.ml.boost.homo_boost import SFXgboost

bst = SFXgboost(server=charlie, clients=[alice, bob])

run SFXgboost

[7]:

bst.train(data, data, params=params, num_boost_round=6)

(_run pid=3817967) 2022-08-19 13:48:05.541675: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/rh/rh-ruby25/root/usr/local/lib64:/opt/rh/rh-ruby25/root/usr/lib64:/opt/rh/devtoolset-11/root/usr/lib64:/opt/rh/devtoolset-11/root/usr/lib:/opt/rh/devtoolset-11/root/usr/lib64/dyninst:/opt/rh/devtoolset-11/root/usr/lib/dyninst
(_run pid=3817967) 2022-08-19 13:48:07,217,217 WARNING [xla_bridge.py:backends:265] No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
(_run pid=3817957) 2022-08-19 13:48:07.943512: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/rh/rh-ruby25/root/usr/local/lib64:/opt/rh/rh-ruby25/root/usr/lib64:/opt/rh/devtoolset-11/root/usr/lib64:/opt/rh/devtoolset-11/root/usr/lib:/opt/rh/devtoolset-11/root/usr/lib64/dyninst:/opt/rh/devtoolset-11/root/usr/lib/dyninst
(pid=3817961) 2022-08-19 13:48:08.108831: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/rh/rh-ruby25/root/usr/local/lib64:/opt/rh/rh-ruby25/root/usr/lib64:/opt/rh/devtoolset-11/root/usr/lib64:/opt/rh/devtoolset-11/root/usr/lib:/opt/rh/devtoolset-11/root/usr/lib64/dyninst:/opt/rh/devtoolset-11/root/usr/lib/dyninst
(pid=3817959) 2022-08-19 13:48:08.068793: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/rh/rh-ruby25/root/usr/local/lib64:/opt/rh/rh-ruby25/root/usr/lib64:/opt/rh/devtoolset-11/root/usr/lib64:/opt/rh/devtoolset-11/root/usr/lib:/opt/rh/devtoolset-11/root/usr/lib64/dyninst:/opt/rh/devtoolset-11/root/usr/lib/dyninst
(pid=3817954) 2022-08-19 13:48:08.108831: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/rh/rh-ruby25/root/usr/local/lib64:/opt/rh/rh-ruby25/root/usr/lib64:/opt/rh/devtoolset-11/root/usr/lib64:/opt/rh/devtoolset-11/root/usr/lib:/opt/rh/devtoolset-11/root/usr/lib64/dyninst:/opt/rh/devtoolset-11/root/usr/lib/dyninst
(pid=3817964) 2022-08-19 13:48:08.108831: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/rh/rh-ruby25/root/usr/local/lib64:/opt/rh/rh-ruby25/root/usr/lib64:/opt/rh/devtoolset-11/root/usr/lib64:/opt/rh/devtoolset-11/root/usr/lib:/opt/rh/devtoolset-11/root/usr/lib64/dyninst:/opt/rh/devtoolset-11/root/usr/lib/dyninst
(pid=3817963) 2022-08-19 13:48:08.111619: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/rh/rh-ruby25/root/usr/local/lib64:/opt/rh/rh-ruby25/root/usr/lib64:/opt/rh/devtoolset-11/root/usr/lib64:/opt/rh/devtoolset-11/root/usr/lib:/opt/rh/devtoolset-11/root/usr/lib64/dyninst:/opt/rh/devtoolset-11/root/usr/lib/dyninst
(pid=3817956) 2022-08-19 13:48:08.108832: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/rh/rh-ruby25/root/usr/local/lib64:/opt/rh/rh-ruby25/root/usr/lib64:/opt/rh/devtoolset-11/root/usr/lib64:/opt/rh/devtoolset-11/root/usr/lib:/opt/rh/devtoolset-11/root/usr/lib64/dyninst:/opt/rh/devtoolset-11/root/usr/lib/dyninst
(pid=3817955) 2022-08-19 13:48:08.127188: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/rh/rh-ruby25/root/usr/local/lib64:/opt/rh/rh-ruby25/root/usr/lib64:/opt/rh/devtoolset-11/root/usr/lib64:/opt/rh/devtoolset-11/root/usr/lib:/opt/rh/devtoolset-11/root/usr/lib64/dyninst:/opt/rh/devtoolset-11/root/usr/lib/dyninst
(pid=3817960) 2022-08-19 13:48:08.157280: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/rh/rh-ruby25/root/usr/local/lib64:/opt/rh/rh-ruby25/root/usr/lib64:/opt/rh/devtoolset-11/root/usr/lib64:/opt/rh/devtoolset-11/root/usr/lib:/opt/rh/devtoolset-11/root/usr/lib64/dyninst:/opt/rh/devtoolset-11/root/usr/lib/dyninst
(pid=3817962) 2022-08-19 13:48:08.127188: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/rh/rh-ruby25/root/usr/local/lib64:/opt/rh/rh-ruby25/root/usr/lib64:/opt/rh/devtoolset-11/root/usr/lib64:/opt/rh/devtoolset-11/root/usr/lib:/opt/rh/devtoolset-11/root/usr/lib64/dyninst:/opt/rh/devtoolset-11/root/usr/lib/dyninst
(pid=3817968) 2022-08-19 13:48:08.140477: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/rh/rh-ruby25/root/usr/local/lib64:/opt/rh/rh-ruby25/root/usr/lib64:/opt/rh/devtoolset-11/root/usr/lib64:/opt/rh/devtoolset-11/root/usr/lib:/opt/rh/devtoolset-11/root/usr/lib64/dyninst:/opt/rh/devtoolset-11/root/usr/lib/dyninst
(_run pid=3817957) 2022-08-19 13:48:09,720,720 WARNING [xla_bridge.py:backends:265] No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
(pid=3817954) /home/fengjun.feng/miniconda3/envs/py3.8/lib/python3.8/site-packages/xgboost/compat.py:36: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
(pid=3817954)   from pandas import MultiIndex, Int64Index
(_run pid=3817964) /home/fengjun.feng/miniconda3/envs/py3.8/lib/python3.8/site-packages/xgboost/compat.py:36: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
(_run pid=3817964)   from pandas import MultiIndex, Int64Index
(pid=3817963) /home/fengjun.feng/miniconda3/envs/py3.8/lib/python3.8/site-packages/xgboost/compat.py:36: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
(pid=3817963)   from pandas import MultiIndex, Int64Index
(HomoBooster pid=3817954) /home/fengjun.feng/miniconda3/envs/py3.8/lib/python3.8/site-packages/xgboost/data.py:262: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
(HomoBooster pid=3817954)   elif isinstance(data.columns, (pd.Int64Index, pd.RangeIndex)):
(HomoBooster pid=3817954) /home/fengjun.feng/miniconda3/envs/py3.8/lib/python3.8/site-packages/xgboost/data.py:262: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
(HomoBooster pid=3817954)   elif isinstance(data.columns, (pd.Int64Index, pd.RangeIndex)):
(HomoBooster pid=3817964) /home/fengjun.feng/miniconda3/envs/py3.8/lib/python3.8/site-packages/xgboost/data.py:262: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
(HomoBooster pid=3817964)   elif isinstance(data.columns, (pd.Int64Index, pd.RangeIndex)):
(HomoBooster pid=3817964) /home/fengjun.feng/miniconda3/envs/py3.8/lib/python3.8/site-packages/xgboost/data.py:262: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
(HomoBooster pid=3817964)   elif isinstance(data.columns, (pd.Int64Index, pd.RangeIndex)):
(HomoBooster pid=3817963) /home/fengjun.feng/miniconda3/envs/py3.8/lib/python3.8/site-packages/xgboost/data.py:262: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
(HomoBooster pid=3817963)   elif isinstance(data.columns, (pd.Int64Index, pd.RangeIndex)):
(HomoBooster pid=3817963) /home/fengjun.feng/miniconda3/envs/py3.8/lib/python3.8/site-packages/xgboost/data.py:262: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
(HomoBooster pid=3817963)   elif isinstance(data.columns, (pd.Int64Index, pd.RangeIndex)):

(HomoBooster pid=3817964) [0] train-merror:0.01366    valid-merror:0.01366
(HomoBooster pid=3817954) [0] train-merror:0.01366    valid-merror:0.01366
(HomoBooster pid=3817963) [0] train-merror:0.01366    valid-merror:0.01366
(HomoBooster pid=3817964) [1] train-merror:0.00820    valid-merror:0.00820
(HomoBooster pid=3817954) [1] train-merror:0.00820    valid-merror:0.00820
(HomoBooster pid=3817963) [1] train-merror:0.00820    valid-merror:0.00820
(HomoBooster pid=3817964) [2] train-merror:0.00820    valid-merror:0.00820
(HomoBooster pid=3817954) [2] train-merror:0.00820    valid-merror:0.00820
(HomoBooster pid=3817963) [2] train-merror:0.00820    valid-merror:0.00820
(HomoBooster pid=3817964) [3] train-merror:0.01093    valid-merror:0.01093
(HomoBooster pid=3817954) [3] train-merror:0.01093    valid-merror:0.01093
(HomoBooster pid=3817963) [3] train-merror:0.01093    valid-merror:0.01093
(HomoBooster pid=3817964) [4] train-merror:0.00820    valid-merror:0.00820
(HomoBooster pid=3817954) [4] train-merror:0.00820    valid-merror:0.00820
(HomoBooster pid=3817963) [4] train-merror:0.00820    valid-merror:0.00820
(HomoBooster pid=3817964) [5] train-merror:0.00546    valid-merror:0.00546
(HomoBooster pid=3817954) [5] train-merror:0.00546    valid-merror:0.00546
(HomoBooster pid=3817963) [5] train-merror:0.00546    valid-merror:0.00546

Now our Federated XGBoost training is complete, where the BST is the federated Boost object

Conclusion#

  • This tutorial introduces how to use tree models for training etc

  • SFXgboost encapsulates the logic of the federated subtree model. Sfxgboost trained models remain compatible with XGBoost, and we can directly use the existing infrastructure for online prediction and so on.

  • Next, you can try SFXgboost on your data, just need to follow this tutorial