Federated learning is a distributed machine learning technique that trains a global model by exchanging model parameters or intermediate results among multiple data sources. Although federated learning achieves physical isolation of data, the local data of federated learning clients are still at risk of leakage under the attack of malicious individuals. For this reason, combining data protection techniques (e.g., differential privacy techniques) with federated learning is a sure way to further improve the data security of federated learning models. In this survey, we review recent advances in the research of differentially-private federated learning models. First, we introduce the workflow of federated learning and the theoretical basis of differential privacy. Then, we review three differentially-private federated learning paradigms: centra l differential privacy, local differential privacy, and distributed differential privacy. After this, we review the algorithmic optimization and communication cost optimization of federated learning models with differential privacy. Finally, we review the applications of federated learning models with differential privacy in various domains. By systematically summarizing the existing research, we propose future research opportunities.
Bishop, C.M. and Nasrabadi, N.M. (2006) Pattern Recognition and Machine Learning. Springer, New York.
Nadkarni, P.M., Ohno-Machado, L. and Chapman, W.W. (2011) Natural Language Processing: An Introduction. Journal of the American Medical Informatics Association, 18, 544-551. https://doi.org/10.1136/amiajnl-2011-000464
Jarvis, R.A. (1983) A Perspective on Range Finding Techniques for Computer Vision. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-5, 122-139. https://doi.org/10.1109/TPAMI.1983.4767365
Fatima, M. and Pasha, M. (2017) Survey of Machine Learning Algorithms for Disease Diagnostic. Journal of Intelligent Learning Systems and Applications, 9, 1-16. https://doi.org/10.4236/jilsa.2017.91001
Bolton, R.J. and Hand, D.J. (2002) Statistical Fraud Detection: A Review. Statistical Science, 17, 235-255. https://doi.org/10.1214/ss/1042727940
Zhao, W., Chellappa, R., Phillips, P.J. and Rosenfeld, A. (2003) Face Recognition: A Literature Survey. ACM Computing Surveys, 35, 399-458. https://doi.org/10.1145/954339.954342
Reddy, D.R. (1976) Speech Recognition by Machine: A Review. Proceedings of the IEEE, 64, 501-531. https://doi.org/10.1109/PROC.1976.10158
Ji, Z., Lipton, Z.C. and Elkan, C. (2014) Differential Privacy and Machine Learning: A Survey and Review.
El Ouadrhiri, A. and Abdelhadi, A. (2022) Differential Privacy for Deep and Federated Learning: A Survey. IEEE Access, 10, 22359-22380. https://doi.org/10.1109/ACCESS.2022.3151670
McMahan, B., Moore, E., Ramage, D., Hampson, S. and Arcas, B.A. (2017) Communication-Efficient Learning of Deep Networks from Decentralized Data. Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, Fort Lauderdale, 20-22 April 2017, 1273-1282.
Ziller, A., Trask, A., Lopardo, A., et al. (2021) Pysyft: A Library for Easy Federated Learning. In: ur Rehman, M.H. and Gaber, M.M., Eds., Federated Learning Systems, Springer, Berlin, 111-139. https://doi.org/10.1007/978-3-030-70604-3_5
Liu, Y., Fan, T., Chen, T., Xu, Q. and Yang, Q. (2021) FATE: An Industrial Grade Platform for Collaborative Learning with Data Protection. Journal of Machine Learning Research, 22, 1-6.
Li, Q., Wen, Z., Wu, Z., et al. (2021) A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection. IEEE Transactions on Knowledge and Data Engineering. https://doi.org/10.1109/TKDE.2021.3124599
Nasr, M., Shokri, R. and Houmansadr, A. (2019) Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-Box Inference Attacks against Centralized and Federated Learning. 2019 IEEE Symposium on Security and Privacy, San Francisco, 19-23 May 2019, 739-753. https://doi.org/10.1109/SP.2019.00065
Li, T., Sahu, A.K., Zaheer, M., Sanjabi, M., Talwalkar, A. and Smith, V. (2020) Federated Optimization in Heterogeneous Networks. Proceedings of Machine Learning and Systems, Vol. 2, 429-450.
Reisizadeh, A., Mokhtari, A., Hassani, H., Jadbabaie, A. and Pedarsani, R. (2020) Fedpaq: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization. International Conference on Artificial Intelligence and Statistics, 26-28 August 2020, 2021-2031.
So, J., Géler, B. and Avestimehr, A.S. (2021) Turbo-Aggregate: Breaking the Quadratic Aggregation Barrier in Secure Federated Learning. IEEE Journal on Selected Areas in Information Theory, 2, 479-489. https://doi.org/10.1109/JSAIT.2021.3054610
Wang, H., Yurochkin, M., Sun, Y., Papailiopoulos, D. and Khazaeni, Y. (2020) Federated Learning with Matched Averaging.
Liu, L., Zhang, J., Song, S.H. and Letaief, K.B. (2020) Client-Edge-Cloud Hierarchical Federated Learning. ICC 2020 IEEE International Conference on Communications, Dublin, 7-11 June 2020, 1-6. https://doi.org/10.1109/ICC40277.2020.9148862
Yang, Q., Liu, Y., Cheng, Y., Kang, Y., Chen, T. and Yu, H. (2019) Federated Learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, Vol. 13, Springer, Berlin, 1-207. https://doi.org/10.1007/978-3-031-01585-4
Ur Rehman, M.H. and Gaber, M.M. (2021) Federated Learning Systems: Towards Next-Generation AI. Springer Nature, Berlin. https://doi.org/10.1007/978-3-030-70604-3
Ludwig, H. and Baracaldo, N. (2022) Federated Learning: A Comprehensive Overview of Methods and Applications. Springer Nature, Berlin. https://doi.org/10.1007/978-3-030-96896-0
Dwork, C., McSherry, F., Nissim, K. and Smith, A. (2006) Calibrating Noise to Sensitivity in Private Data Analysis. In: Theory of Cryptography Conference, Springer, Berlin, 265-284. https://doi.org/10.1007/11681878_14
McSherry, F. and Talwar, K. (2007) Mechanism Design via Differential Privacy. 48th Annual IEEE Symposium on Foundations of Computer Science, Providence, 21-23 October 2007, 94-103. https://doi.org/10.1109/FOCS.2007.66
Nikolov, A., Talwar, K. and Zhang, L. (2013) The Geometry of Differential Privacy: The Sparse and Approximate Cases. Proceedings of the 45th Annual ACM Symposium on Theory of Computing, Palo Alto, 2-4 June 2013, 351-360. https://doi.org/10.1145/2488608.2488652
Dwork, C. (2008) Differential Privacy: A Survey of Results. International Conference on Theory and Applications of Models of Computation, Xi’an, 25-29 April 2008, 1-19. https://doi.org/10.1007/978-3-540-79228-4_1
Dwork, C. and Roth, A. (2014) The Algorithmic Foundations of Differential Privacy. Foundations and Trends in Theoretical Computer Science, 9, 211-407. https://doi.org/10.1561/0400000042
Geyer, R.C., Klein, T. and Nabi, M. (2017) Differentially Private Federated Learning: A Client Level Perspective.
Triastcyn, A. and Faltings, B. (2019) Federated Learning with Bayesian Differential Privacy. 2019 IEEE International Conference on Big Data, Los Angeles, 9-12 December 2019, 2587-2596. https://doi.org/10.1109/BigData47090.2019.9005465
Wei, K., Li, J., Ding, M., Ma, C., et al. (2020) Federated Learning with Differential Privacy: Algorithms and Performance Analysis. IEEE Transactions on Information Forensics and Security, 15, 3454-3469. https://doi.org/10.1109/TIFS.2020.2988575
Bernau, D., Robl, J., Grassal, P.W., Schneider, S. and Kerschbaum, F. (2021) Comparing Local and Central Differential Privacy Using Membership Inference Attacks. IFIP Annual Conference on Data and Applications Security and Privacy, Calgary, 19-20 July 2021, 22-42. https://doi.org/10.1007/978-3-030-81242-3_2
Zhang, X., Chen, X., Hong, M., Wu, S. and Yi, J. (2022) Understanding Clipping for Federated Learning: Convergence and Client-Level Differential Privacy. International Conference on Machine Learning, Baltimore, 17-23 July 2022, 26048-26067.
Hu, R., Gong, Y. and Guo, Y. (2022) Federated Learning with Sparsified Model Perturbation: Improving Accuracy under Client-Level Differential Privacy.
Li, Z., Huang, Z., Chen, C. and Hong, C. (2019) Quantification of the Leakage in Federated Learning.
Melis, L., Song, C., De Cristofaro, E. and Shmatikov, V. (2019) Exploiting Unintended Feature Leakage in Collaborative Learning. 2019 IEEE Symposium on Security and Privacy, San Francisco, 19-23 May 2019, 691-706. https://doi.org/10.1109/SP.2019.00029
Kasiviswanathan, S.P., Lee, H.K., Nissim, K., Raskhodnikova, S. and Smith, A. (2011) What Can We Learn Privately? SIAM Journal on Computing, 40, 793-826. https://doi.org/10.1137/090756090
Erlingsson, ú., Pihur, V., Korolova, A. (2014) Rappor: Randomized Aggregatable Privacy-Preserving Ordinal Response. Proceedings of the 2014 ACM SIGSAC Conference on Computer and Communications Security, Scottsdale, 3-7 November 2014, 1054-1067. https://doi.org/10.1145/2660267.2660348
Liu, F. (2018) Generalized Gaussian Mechanism for Differential Privacy. IEEE Transactions on Knowledge and Data Engineering, 31, 747-756. https://doi.org/10.1109/TKDE.2018.2845388
Truex, S., Liu, L., Chow, K.H., Gursoy, M.E. and Wei, W. (2020) LDP-Fed: Federated Learning with Local Differential Privacy. Proceedings of the 3rd ACM International Workshop on Edge Systems, Analytics and Networking, Heraklion, 27 April 2020, 61-66. https://doi.org/10.1145/3378679.3394533
Sun, Z., Kairouz, P., Suresh, A.T. and McMahan, H.B. (2019) Can You Really Backdoor Federated Learning?
Xu, M., Ding, B., Wang, T. and Zhou, J. (2020) Collecting and Analyzing Data Jointly from Multiple Services under Local Differential Privacy. Proceedings of the VLDB Endowment, 13, 2760-2772. https://doi.org/10.14778/3407790.3407859
Naseri, M., Hayes, J. and De Cristofaro, E. (2020) Toward Robustness and Privacy in Federated Learning: Experimenting with Local and Central Differential Privacy.
Wang, Y., Tong, Y. and Shi, D. (2020) Federated Latent Dirichlet Allocation: A Local Differential Privacy based Framework. Proceedings of the AAAI Conference on Artificial Intelligence, 34, 6283-6290. https://doi.org/10.1609/aaai.v34i04.6096
Girgis, A., Data, D. and Diggavi, S. (2021) Renyi Differential Privacy of the Subsampled Shuffle Model in Distributed Learning. Advances in Neural Information Processing Systems, Vol. 34, 29181-29192. https://doi.org/10.1145/3460120.3484794
Wei, K., Li, J., Ding, M., Ma, C., Su, H., Zhang, B. and Poor, H.V. (2022) User-Level Privacy-Preserving Federated Learning: Analysis and Performance Optimization. IEEE Transactions on Mobile Computing, 21, 3388-3401. https://doi.org/10.1109/TMC.2021.3056991
Zhou, H., Yang, G., Dai, H. and Liu, G. (2022) PFLF: Privacy-Preserving Federated Learning Framework for Edge Computing. IEEE Transactions on Information Forensics and Security, 17, 1905-1918. https://doi.org/10.1109/TIFS.2022.3174394
Thapa, C., Arachchige, P.C.M., Camtepe, S. and Sun, L. (2022) Splitfed: When Federated Learning Meets Split Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 36, 8485-8493. https://doi.org/10.1609/aaai.v36i8.20825
Wu, C., Wu, F., Lyu, L., Qi, T., Huang, Y. and Xie, X. (2022) A Federated Graph Neural Network Framework for Privacy-Preserving Personalization. Nature Communications, 13, Article No. 3091. https://doi.org/10.1038/s41467-022-30714-9
Wang, C., Wu, X., Liu, G., Deng, T., Peng, K. and Wan, S. (2022) Safeguarding Cross-Silo Federated Learning with Local Differential Privacy. Digital Communications and Networks, 8, 446-454. https://doi.org/10.1016/j.dcan.2021.11.006
Zheng, Q., Chen, S., Long, Q. and Su, W. (2021) Federated F-Differential Privacy. International Conference on Artificial Intelligence and Statistics, 13-15 April 2021, 2251-2259.
Bonawitz, K., Ivanov, V., Kreuter, B., et al. (2017) Practical Secure Aggregation for Privacy-Preserving Machine Learning. Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, Dallas, 30 October-3 November 2017, 1175-1191. https://doi.org/10.1145/3133956.3133982
Dwork, C., Kenthapadi, K., McSherry, F., Mironov, I. and Naor, M. (2006) Our Data, Ourselves: Privacy via Distributed Noise Generation. Annual International Conference on the Theory and Applications of Cryptographic Techniques, St. Petersburg, 28 May-1 June 2006, 486-503. https://doi.org/10.1007/11761679_29
Agarwal, N., Suresh, A.T., Yu, F.X.X., Kumar, S. and McMahan, B. (2018) cpSGD: Communication-Efficient and Differentially-Private Distributed SGD. Advances in Neural Information Processing Systems, Vol. 31, 1-12.
Canonne, C.L., Kamath, G. and Steinke, T. (2020) The Discrete Gaussian for Differential Privacy. Advances in Neural Information Processing Systems, Vol. 33, 15676-15688.
Kairouz, P., Liu, Z. and Steinke, T. (2021) The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregation. International Conference on Machine Learning, 18-24 July 2021, 5201-5212.
Agarwal, N., Kairouz, P. and Liu, Z. (2021) The Skellam Mechanism for Differentially Private Federated Learning. Advances in Neural Information Processing Systems, Vol. 34, 5052-5064.
Bao, E., Zhu, Y., Xiao, X., Yang, Y., Ooi, B.C., Tan, B.H.M. and Aung, K.M.M. (2022) Skellam Mixture Mechanism: A Novel Approach to Federated Learning with Differential Privacy. Proceedings of the VLDB Endowment, 15, 2348-2360. https://doi.org/10.14778/3551793.3551798
Chen, W.N., Ozgur, A. and Kairouz, P. (2022) The Poisson Binomial Mechanism for Unbiased Federated Learning with Secure Aggregation. International Conference on Machine Learning, Baltimore, 17-23 July 2022, 3490-3506.
Chen, W.N., Choo, C.A.C., Kairouz, P. and Suresh, A.T. (2022) The Fundamental Price of Secure Aggregation in Differentially Private Federated Learning. International Conference on Machine Learning, Baltimore, 17-23 July 2022, 3056-3089.
Cheu, A., Smith, A., Ullman, J., Zeber, D. and Zhilyaev, M. (2019) Distributed Differential Privacy via Shuffling. Annual International Conference on the Theory and Applications of Cryptographic Techniques, Darmstadt, 19-23 May 2019, 375-403. https://doi.org/10.1007/978-3-030-17653-2_13
Jiang, Z., Wang, W. and Chen, R. (2022) Taming Client Dropout for Distributed Differential Privacy in Federated Learning.
Zhou, Y. and Tang, S. (2020) Differentially Private Distributed Learning. INFORMS Journal on Computing, 32, 779-789. https://doi.org/10.1287/ijoc.2019.0912
Hu, R., Guo, Y., Li, H., Pei, Q. and Gong, Y. (2020) Personalized Federated Learning with Differential Privacy. IEEE Internet of Things Journal, 7, 9530-9539. https://doi.org/10.1109/JIOT.2020.2991416
Van Dijk, M., Nguyen, N.V., Nguyen, T.N., Nguyen, L.M., Tran-Dinh, Q. and Nguyen, P.H. (2020) Asynchronous Federated Learning with Reduced Number of Rounds and with Differential Privacy from Less Aggregated Gaussian Noise.
Girgis, A., Data, D., Diggavi, S., Kairouz, P. and Suresh, A.T. (2021) Shuffled Model of Differential Privacy in Federated Learning. International Conference on Artificial Intelligence and Statistics, 13-15 April 2021, 2521-2529.
Zhang, L., Zhu, T., Xiong, P., Zhou, W. and Yu, P. (2022) A Robust Game-Theoretical Federated Learning Framework with Joint Differential Privacy. IEEE Transactions on Knowledge and Data Engineering. https://doi.org/10.1109/TKDE.2021.3140131
Denisov, S., McMahan, B., Rush, K., Smith, A. and Thakurta, A. (2022) Improved Differential Privacy for SGD via Optimal Private Linear Operators on Adaptive Streams. Advances in Neural Information Processing Systems.
Lian, Z., Wang, W. and Su, C. (2021) COFEL: Communication-efficient and Optimized Federated Learning with Local Differential Privacy. ICC 2021-IEEE International Conference on Communications, Montreal, 14-23 June 2021, 1-6. https://doi.org/10.1109/ICC42927.2021.9500632
Amiri, S., Belloum, A., Klous, S. and Gommans, L. (2021) Compressive Differentially Private Federated Learning through Universal Vector Quantization. AAAI Workshop on Privacy-Preserving Artificial Intelligence, 2-9 February 2021, 1-5.
Liu, J., Lou, J., Xiong, L., Liu, J. and Meng, X. (2021) Projected Federated Averaging with Heterogeneous Differential Privacy. Proceedings of the VLDB Endowment, 15, 828-840. https://doi.org/10.14778/3503585.3503592
Truex, S., Baracaldo, N., Anwar, A., Steinke, T., Ludwig, H., Zhang, R. and Zhou, Y. (2019) A Hybrid Approach to Privacy-Preserving Federated Learning. Proceedings of the 12th ACM Workshop on Artificial Intelligence and Security, London, 15 November 2019, 1-11. https://doi.org/10.1145/3338501.3357370
Andr’s, M.E., Bordenabe, N.E., Chatzikokolakis, K. and Palamidessi, C. (2013) Geo-Indistinguishability: Differential Privacy for Location-Based Systems. Proceedings of the 2013 ACM SIGSAC Conference on Computer & Communications Security, Berlin, 4-8 November 2013, 901-914. https://doi.org/10.1145/2508859.2516735
Wang, S., Huang, L., Nie, Y., Zhang, X., Wang, P., Xu, H. and Yang, W. (2019) Local Differential Private Data Aggregation for Discrete Distribution Estimation. IEEE Transactions on Parallel and Distributed Systems, 30, 2046-2059. https://doi.org/10.1109/TPDS.2019.2899097
Zhao, Y., Zhao, J., Yang, M., et al. (2020) Local Differential Privacy-Based Federated Learning for Internet of Things. IEEE Internet of Things Journal, 8, 8836-8853. https://doi.org/10.1109/JIOT.2020.3037194
Cao, H., Liu, S., Zhao, R. and Xiong, X. (2020) IFed: A Novel Federated Learning Framework for Local Differential Privacy in Power Internet of Things. International Journal of Distributed Sensor Networks, 16. https://doi.org/10.1177/1550147720919698
Jia, B., Zhang, X., Liu, J., Zhang, Y., Huang, K. and Liang, Y. (2021) Blockchain-Enabled Federated Learning Data Protection Aggregation Scheme with Differential Privacy and Homomorphic Encryption in IIoT. IEEE Transactions on Industrial Informatics, 18, 4049-4058. https://doi.org/10.1109/TII.2021.3085960
Olowononi, F.O., Rawat, D.B. and Liu, C. (2021) Federated Learning with Differential Privacy for Resilient Vehicular Cyber Physical Systems. 2021 IEEE 18th Annual Consumer Communications & Networking Conference (CCNC), Las Vegas, 9-12 January 2021, 1-5. https://doi.org/10.1109/CCNC49032.2021.9369480
Liu, Q., Chen, C., Qin, J., Dou, Q. and Heng, P.A. (2021) Feddg: Federated Domain Generalization on Medical Image Segmentation via Episodic Learning in Continuous Frequency Space. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, 20-25 June 2021, 1013-1023. https://doi.org/10.1109/CVPR46437.2021.00107
Kaissis, G., Ziller, A., Passerat-Palmbach, J., Ryffel, T., et al. (2021) End-to-End Privacy Preserving Deep Learning on Multi-Institutional Medical Imaging. Nature Machine Intelligence, 3, 473-484. https://doi.org/10.1038/s42256-021-00337-8
Adnan, M., Kalra, S., Cresswell, J.C., Taylor, G.W. and Tizhoosh, H.R. (2022) Federated Learning and Differential Privacy for Medical Image Analysis. Scientific Reports, 12, Article No. 1953. https://doi.org/10.1038/s41598-022-05539-7
Zhang, Z., Zhang, L., Li, Q., Wang, K., He, N. and Gao, T. (2022) Privacy-Enhanced Momentum Federated Learning via Differential Privacy and Chaotic System in Industrial Cyber-Physical Systems. ISA Transactions, 128, 17-31. https://doi.org/10.1016/j.isatra.2021.09.007
Liu, W., Cheng, J., Wang, X., Lu, X. and Yin, J. (2022) Hybrid Differential Privacy based Federated Learning for Internet of Things. Journal of Systems Architecture, 124, Article ID: 102418. https://doi.org/10.1016/j.sysarc.2022.102418
Cormode, G., Jha, S., Kulkarni, T., Li, N., Srivastava, D. and Wang, T. (2018) Privacy at Scale: Local Differential Privacy in Practice. Proceedings of the 2018 International Conference on Management of Data, Houston, 10-15 June 2018, 1655-1658. https://doi.org/10.1145/3183713.3197390