{
    "cells": [
        {
            "cell_type": "markdown",
            "id": "d69d2c34",
            "metadata": {},
            "source": [
                "# Secure Aggregation"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "e452d1f9",
            "metadata": {},
            "source": [
                ">The following codes are demos only. It's **NOT for production** due to system security concerns, please **DO NOT** use it directly in production."
            ]
        },
        {
            "cell_type": "markdown",
            "id": "3bf12bc1",
            "metadata": {},
            "source": [
                "\n",
                "It is recommended to use [jupyter](https://jupyter.org/) to run this tutorial."
            ]
        },
        {
            "cell_type": "markdown",
            "id": "ce637185-bda7-44bb-aa79-b6a5b21ec851",
            "metadata": {},
            "source": [
                "Secure aggregation can be expressed as multiple parties owning data, and cooperating to complete the computation of aggregated values (such as summation) without revealing their private data.\n",
                "\n",
                "Secure aggregation is an important concept in federated learning. There have been many studies in the academic community. SecretFlow has used secure aggregation in horizontal federated gradient/weight aggregation and data statistics (such as data exploration and preprocessing).\n",
                "\n",
                "The following explains the secure aggregation used by secretflow.\n"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "d52a334d-96c8-4801-b5c1-b12710d469ab",
            "metadata": {},
            "source": [
                "## Preparation\n",
                "\n",
                "Initialize SecretFlow."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "id": "dff940f3-f88c-4e6e-86f5-0bd00f15e14c",
            "metadata": {},
            "outputs": [],
            "source": [
                "import secretflow as sf\n",
                "\n",
                "# In case you have a running SecretFlow runtime already.\n",
                "sf.shutdown()\n",
                "\n",
                "sf.init(['alice', 'bob'], address='local')"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "6fd12c4f-c07c-4218-80ee-b6757482ea40",
            "metadata": {},
            "source": [
                "Prepare some data for testing."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 2,
            "id": "0bf50097-f1dd-48e9-86f3-3d353ffa2d47",
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "arr0:\n",
                        " [[0.53867365 0.69040348 0.42628929]\n",
                        " [0.76128941 0.5444343  0.7680543 ]] \n",
                        "arr1:\n",
                        " [[0.74303296 0.7274792  0.47244091]\n",
                        " [0.88295957 0.80091356 0.82681861]]\n",
                        "Sum:\n",
                        " [[1.28170662 1.41788268 0.8987302 ]\n",
                        " [1.64424898 1.34534786 1.59487291]]\n",
                        "Average:\n",
                        " [[0.64085331 0.70894134 0.4493651 ]\n",
                        " [0.82212449 0.67267393 0.79743646]]\n",
                        "Min:\n",
                        " [[0.53867365 0.69040348 0.42628929]\n",
                        " [0.76128941 0.5444343  0.7680543 ]]\n",
                        "Max:\n",
                        " [[0.74303296 0.7274792  0.47244091]\n",
                        " [0.88295957 0.80091356 0.82681861]]\n"
                    ]
                }
            ],
            "source": [
                "import numpy as np\n",
                "\n",
                "arr0, arr1 = np.random.rand(2, 3), np.random.rand(2, 3)\n",
                "print('arr0:\\n', arr0, '\\narr1:\\n', arr1)\n",
                "\n",
                "print('Sum:\\n', np.sum([arr0, arr1], axis=0))\n",
                "print('Average:\\n', np.average([arr0, arr1], axis=0))\n",
                "print('Min:\\n', np.min([arr0, arr1], axis=0))\n",
                "print('Max:\\n', np.max([arr0, arr1], axis=0))"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "e3759e41-7ee6-4225-a504-969990b73338",
            "metadata": {},
            "source": [
                "Create parties alice and bob."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 3,
            "id": "e7d060f4-a20a-404e-821b-e93e45b8ced3",
            "metadata": {},
            "outputs": [],
            "source": [
                "alice, bob = sf.PYU('alice'), sf.PYU('bob')"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "4a23a3bf-3003-4c7a-a1e9-3bca5fba2679",
            "metadata": {},
            "source": [
                "## Aggregate operation\n",
                "\n",
                "SecretFlow provides a variety of ```Aggregator``` for users to choose from, each ```Aggregator``` provides the function of sum/average.\n",
                "\n",
                "### SPU based security aggregation\n",
                "\n",
                "[SPU](../design/spu.md) is a security device in SecretFlow, and its underlying principle is [MPC](https://en.wikipedia.org/wiki/Secure_multi-party_computation). The SecretFlow implements SPU-based secure aggregation, and the following shows how to use it."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 4,
            "id": "5209d8db-547a-4a2f-a02d-85515554241c",
            "metadata": {},
            "outputs": [],
            "source": [
                "# Create an spu device.\n",
                "spu = sf.SPU(sf.utils.testing.cluster_def(['alice', 'bob']))\n",
                "\n",
                "# Create an aggregator instance using this spu.\n",
                "spu_aggr = sf.security.aggregation.SPUAggregator(spu)"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 5,
            "id": "ec966557-a996-4a0f-8c65-27989770b849",
            "metadata": {},
            "outputs": [],
            "source": [
                "# Simulate that alice and bob hold data respectively\n",
                "a = alice(lambda: arr0)()\n",
                "b = bob(lambda: arr1)()"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 6,
            "id": "cd1ed90a-efed-4022-847b-969a5d7ccb63",
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/plain": [
                            "array([[1.2817066 , 1.4178827 , 0.89873016],\n",
                            "       [1.644249  , 1.3453479 , 1.594873  ]], dtype=float32)"
                        ]
                    },
                    "execution_count": 6,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "# Sum the data.\n",
                "sf.reveal(spu_aggr.sum([a, b], axis=0))"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 7,
            "id": "c052616f-f9ca-4ae0-af16-db203110e874",
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/plain": [
                            "array([[0.6408533 , 0.70894134, 0.44936508],\n",
                            "       [0.8221245 , 0.67267394, 0.7974364 ]], dtype=float32)"
                        ]
                    },
                    "execution_count": 7,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "# Average the data.\n",
                "sf.reveal(spu_aggr.average([a, b], axis=0))"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "bef5c70c-e892-4d2e-806d-d2ec1c37f02c",
            "metadata": {},
            "source": [
                "### Masking with One-Time Pads\n",
                "\n",
                "`Masking with One-Time Pads` negotiates a secret for every two participants, then uses the secret to hide its input $x$, and each participant outputs.\n",
                "\n",
                "$$ y_u = x_u + \\sum_{u < v}s_{u,v} - \\sum_{u > v}s_{u,v}\\ mod\\ R  $$\n",
                "\n",
                "the secrets are cancelled out after aggregation and then we can get the correct result.\n",
                "\n",
                "$$ \\sum y = \\sum x $$\n",
                "\n",
                "\n",
                "For example, the participants Alice, Bob, and Carol each own $x_1, x_2, x_3$, negotiate the secret $s_{a,b}, s_{a,c}, s_{b,c}$, and then output:\n",
                "$y_1 = x_1 + s_{a,b} + s_{a,c}$  \n",
                "$y_2 = x_2 - s_{a,b} + s_{b,c}$  \n",
                "$y_3 = x_3 - s_{a,c} - s_{b,c}$  \n",
                "then it is easy to get  $$ y_1 + y_2 + y_3 = x_1 + s_{a,b} + s_{a,c} + x_2 - s_{a,b} + s_{b,c} + x_3 - s_{a,c} - s_{b,c} = x_1 + x_2 + x_3 $$\n",
                "\n",
                "Note that `Masking with One-Time Pads` is based on semi-honest assumptions and does not support client dropping. For more information, please refer to [Practical Secure Aggregation\n",
                "for Privacy-Preserving Machine Learning](https://eprint.iacr.org/2017/281.pdf)\n",
                "\n",
                "> **_Warning:_** The SecureAggregator uses [numpy.random.PCG64](https://numpy.org/doc/stable/reference/random/bit_generators/pcg64.html#numpy.random.PCG64). There are many discussions of whether PCG is a CSPRNG (e.g. https://crypto.stackexchange.com/questions/77101/is-the-pcg-prng-a-csprng-or-why-not), we prefer a conservative strategy unless a further security analysis came up. Therefore we recommend users to use a standardized CSPRNG in industrial scenarios."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 8,
            "id": "874c750c-8c2f-4ba5-a427-eecab2ca6f3c",
            "metadata": {},
            "outputs": [],
            "source": [
                "# Create a secure aggregator instance with alice and bob,\n",
                "# where alice is responsible for performing aggregate computing operations.\n",
                "secure_aggr = sf.security.aggregation.SecureAggregator(device=alice, participants=[alice, bob])"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 9,
            "id": "f1c6dbb8-b890-4d8f-af97-09068e3bf9ce",
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/plain": [
                            "array([[1.28170395, 1.41788101, 0.89872742],\n",
                            "       [1.64424515, 1.34534454, 1.59486771]])"
                        ]
                    },
                    "execution_count": 9,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "# Sum the data.\n",
                "sf.reveal(secure_aggr.sum([a, b], axis=0))"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 10,
            "id": "e1907fd5-e26a-4688-b631-cc8ac7b3f838",
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/plain": [
                            "array([[0.64085197, 0.70894051, 0.44936371],\n",
                            "       [0.82212257, 0.67267227, 0.79743385]])"
                        ]
                    },
                    "execution_count": 10,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "# Average the data.\n",
                "sf.reveal(secure_aggr.average([a, b], axis=0))"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "60410eb4-0137-43cd-86fd-9d4fab394108",
            "metadata": {},
            "source": [
                "### Plaintext aggregation (for test only, not recommended for production use)\n",
                "\n",
                "**`PlainAggregator` is used for test only and not for production use.**\n",
                "\n",
                "For simple local simulation, SecretFlow also provides a plaintext aggregator."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 12,
            "id": "24f57e1d-c6de-48d4-a2ac-5e994b45b62c",
            "metadata": {},
            "outputs": [],
            "source": [
                "# Create a plaintext aggregator instance and alice is responsible for performing aggregation.\n",
                "plain_aggr = sf.security.aggregation.PlainAggregator(alice)"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 13,
            "id": "767c4beb-c416-4a69-ba5a-15ef1ffdd59f",
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/plain": [
                            "array([[1.2817066 , 1.4178827 , 0.89873016],\n",
                            "       [1.644249  , 1.3453479 , 1.594873  ]], dtype=float32)"
                        ]
                    },
                    "execution_count": 13,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "# Sum the data.\n",
                "sf.reveal(plain_aggr.sum([a, b], axis=0))"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 14,
            "id": "6046af97-6b2a-4945-b85d-a95bf80a9344",
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/plain": [
                            "array([[0.6408533 , 0.70894134, 0.44936508],\n",
                            "       [0.8221245 , 0.67267394, 0.7974365 ]], dtype=float32)"
                        ]
                    },
                    "execution_count": 14,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "# Average the data.\n",
                "sf.reveal(plain_aggr.average([a, b], axis=0))"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "ccb4e3cc-b693-432a-b6a6-183d3da40195",
            "metadata": {},
            "source": [
                "## Comparison\n",
                "\n",
                "In addition, SecretFlow provides a variety of ``Comparator```, providing operations such as maximum (max)/minimum (min).\n",
                "For example, in horizontal partitioned data scenario, global values can be obtained through secure comparison without exposing the private information of the participants."
            ]
        },
        {
            "cell_type": "markdown",
            "id": "c45ae947-d550-4868-b58a-6d7eafacfc24",
            "metadata": {},
            "source": [
                "### SPU based security comparison\n",
                "\n",
                "SecretFlow implements SPU-based secure comparison, and the following shows how to use it."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 15,
            "id": "f3a4ea28-2ac0-468f-addc-527c8b1e3273",
            "metadata": {},
            "outputs": [],
            "source": [
                "# Create an spu comparator instance.\n",
                "spu_com = sf.security.compare.SPUComparator(spu)"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 16,
            "id": "59f369d3-75dc-4730-8aed-3efc3d31cedb",
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/plain": [
                            "array([[0.53867364, 0.69040346, 0.4262893 ],\n",
                            "       [0.7612894 , 0.5444343 , 0.7680543 ]], dtype=float32)"
                        ]
                    },
                    "execution_count": 16,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "# Get the minimum.\n",
                "sf.reveal(spu_com.min([a, b], axis=0))"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 18,
            "id": "ecb25b3d-c2cc-45a1-863e-d171bc446eb9",
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/plain": [
                            "array([[0.743033  , 0.7274792 , 0.4724409 ],\n",
                            "       [0.88295954, 0.8009136 , 0.8268186 ]], dtype=float32)"
                        ]
                    },
                    "execution_count": 18,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "# Get the maximum.\n",
                "sf.reveal(spu_com.max([a, b], axis=0))"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "c5090d29-bf15-460a-b941-56bbcffac1f8",
            "metadata": {},
            "source": [
                "### Plaintext comparison (not recommended for production use)\n",
                "\n",
                "**`PlainComparator` is used for test only and not for production use.**\n",
                "\n",
                "For simple local simulation, SecretFlow also provides a plaintext comparator."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 19,
            "id": "b26ffc58-66a9-4a59-a4c6-5387ed50189d",
            "metadata": {},
            "outputs": [],
            "source": [
                "# Create a plaintext comparator instance and alice is responsible for performing the comparison.\n",
                "plain_com = sf.security.compare.PlainComparator(alice)"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 21,
            "id": "536bc777-9d5d-442a-ab33-d2215b8bb930",
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/plain": [
                            "array([[0.53867364, 0.69040346, 0.4262893 ],\n",
                            "       [0.7612894 , 0.5444343 , 0.7680543 ]], dtype=float32)"
                        ]
                    },
                    "execution_count": 21,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "# Get the minimum.\n",
                "sf.reveal(plain_com.min([a, b], axis=0))"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 22,
            "id": "f60702d6-7fa1-49d1-8dc2-08fca911925f",
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/plain": [
                            "array([[0.743033  , 0.7274792 , 0.4724409 ],\n",
                            "       [0.88295954, 0.8009136 , 0.8268186 ]], dtype=float32)"
                        ]
                    },
                    "execution_count": 22,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "# Get the maximum.\n",
                "sf.reveal(plain_com.max([a, b], axis=0))"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "e6e45347",
            "metadata": {},
            "source": [
                "## Ending"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 23,
            "id": "2eb49d63",
            "metadata": {},
            "outputs": [],
            "source": [
                "sf.shutdown()"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "1bc9bd95-5e58-4361-9c5f-0893edc242cd",
            "metadata": {},
            "source": [
                "## Summarize\n",
                "\n",
                "This article shows the security aggregation of SecretFlow. SecretFlow provides a variety of security aggregation, and users can implement different security policies according to their own needs.\n",
                "For the plaintext aggregation and compare, it is not recommended to use it in the production environment."
            ]
        }
    ],
    "metadata": {
        "kernelspec": {
            "display_name": "Python 3.8.13 ('py3.8')",
            "language": "python",
            "name": "python3"
        },
        "language_info": {
            "codemirror_mode": {
                "name": "ipython",
                "version": 3
            },
            "file_extension": ".py",
            "mimetype": "text/x-python",
            "name": "python",
            "nbconvert_exporter": "python",
            "pygments_lexer": "ipython3",
            "version": "3.8.13"
        },
        "vscode": {
            "interpreter": {
                "hash": "66d1547304beaba725027c44e57cc46fc747862fe9496520910412a3187eb35f"
            }
        }
    },
    "nbformat": 4,
    "nbformat_minor": 5
}