Particle accelerators play an important role in a wide range of scientific discoveries and industrial applications. The self-consistent multi-particle simulation based on the particle-in-cell (PIC) method has been used to study charged particle beam dynamics inside those accelerators. However, the PIC simulation is time-consuming and needs to use modern parallel computers for high-resolution applications. In this paper, we implemented a parallel beam dynamics PIC code on multi-node hybrid architecture computers with multiple Graphics Processing Units (GPUs). We used two methods to parallelize the PIC code on multiple GPUs and observed that the replication method is a better choice for moderate problem size and current computer hardware while the domain decomposition method might be a better choice for large problem size and more advanced computer hardware that allows direct communications among multiple GPUs. Using the multi-node hybrid architectures at Oak Ridge Leadership Computing Facility (OLCF), the optimized GPU PIC code achieves a reasonable parallel performance and scales up to 64 GPUs with 16 million particles.
Birdsall, C.K. (1991) Particle-in-Cell Charged-Particle Simulations, Plus Monte Carlo Collisions with Neutral Atoms, PIC-MCC. IEEE Transactions on Plasma Science, 19, 65-85. https://doi.org/10.1109/27.106800
Friedman, A., Grote, D.P. and Haber, I. (1992) Three-Dimensional Particle Simulation of Heavy-Ion Fusion Beams. Physics of Plasmas, 4, 2203-2210. https://doi.org/10.1063/1.860024
Qiang, J., Ryne, R.D., Habib, S. and Decyk, V. (2000) An Object-Oriented Parallel Particle-in-Cell Code for Beam Dynamics Simulation in Linear Accelerators. Journal of Computational Physics, 163, 434-451. https://doi.org/10.1006/jcph.2000.6570 http://www.sciencedirect.com/science/article/pii/S0021999100965707
Qiang, J., Furman, M.A. and Ryne, R.D. (2004) A Parallel Particle-in-Cell Model for Beam: Beam Interaction in High Energy Ring Colliders. Journal of Computational Physics, 198, 278-294. https://doi.org/10.1016/j.jcp.2004.01.008
Amundson, J., Spentzouris, P., Qiang, J. and Ryne, R. (2006) Synergia: An Accelerator Modeling Tool with 3-D Space Charge. Journal of Computational Physics, 211, 229-248. http://www.sciencedirect.com/science/article/pii/S0021999105002718 https://doi.org/10.1016/j.jcp.2005.05.024
Qiang, J., Lidia, S., Ryne, R.D. and Limborg-Deprey, C. (2006) A Three-Dimensional Quasi-Static Model for High Brightness Beam Dynamics Simulation. Physical Review Accelerators and Beams, 9, Article ID: 044204. https://doi.org/10.1103/PhysRevSTAB.9.044204
Uriot, D. and Pichoff, N. (2015) Tracewin, CEA Saclay.
Batygin, Y.K. (2005) Particle-in-Cell Code BEAMPATH for Beam Dynamics Simulations in Linear Accelerators and Beamlines. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 539, 455-489. https://doi.org/10.1016/j.nima.2004.10.029
Pharr, M. and Fernando, R. (2005) GPU Gems 2: Programming Techniques for High-Performance Graphics and General-Purpose Computation. Addison-Wesley Professional, Boston, MA.
Owens, J.D., Luebke, D., Govindaraju, N., Harris, M., Krüger, J., Lefohn, A.E. and Purcell, T.J. (2007) A Survey of General-Purpose Computation on Graphics Hardware. Computer Graphics Forum, 26, 80-113. https://doi.org/10.1111/j.1467-8659.2007.01012.x
Owens, J.D., Houston, M., Luebke, D., Green, S., Stone, J.E. and Phillips, J.C. (2008) GPU Computing. Proceedings of the IEEE, 96, 879-899. https://doi.org/10.1109/JPROC.2008.917757
Sanders, J. and Kandrot, E. (2010) CUDA by Example: An Introduction to General-Purpose GPU Programming. Portable Documents, Addison-Wesley Professional, Boston, MA.
Stantchev, G., Dorland, W. and Gumerov, N. (2008) Fast Parallel Particle-to-Grid Interpolation for Plasma PIC Simulations on the GPU. Journal of Parallel and Distributed Computing, 68, 1339-1349. https://doi.org/10.1016/j.jpdc.2008.05.009
Burau, H., Widera, R., Honig, W., Juckeland, G., Debus, A., Kluge, T., Schramm, U., Cowan, T.E., Sauerbrey, R. and Bussmann, M. (2010) PIConGPU: A Fully Relativistic Particle-in-Cell Code for a GPU Cluster. IEEE Transactions on Plasma Science, 38, 2831-2839. https://doi.org/10.1109/TPS.2010.2064310
Decyk, V.K. and Singh, T.V. (2011) Adaptable Particle-in-Cell Algorithms for Graphical Processing Units. Computer Physics Communications, 182, 641-648. https://doi.org/10.1016/j.cpc.2010.11.009
Joseph, R.G., Ravunnikutty, G., Ranka, S., D'Azevedo, E. and Klasky, S. (2011) Efficient GPU Implementation for Particle in Cell Algorithm. 2011 IEEE International Parallel & Distributed Processing Symposium, Anchorage, AK, 16-20 May 2011, 395-406. https://doi.org/10.1109/IPDPS.2011.46
Rossi, F., Londrillo, P., Sgattoni, A., Sinigardi, S. and Turchetti, G. (2012) Towards Robust Algorithms for Current Deposition and Dynamic Load-Balancing in a GPU Particle in Cell Code. AIP Conference Proceedings, 1507, 184-192. https://doi.org/10.1063/1.4773692
Bastrakov, S., Donchenko, R., Gonoskov, A., Efimenko, E., Malyshev, A., Meyerov, I. and Surmin, I. (2012) Particle-in-Cell Plasma Simulation on Heterogeneous Cluster Systems. Journal of Computational Science, 3, 474-479. https://doi.org/10.1016/j.jocs.2012.08.012
Chen, G., Chacon, L. and Barnes, D.C. (2012) An Efficient Mixed-Precision, Hybrid CPU-GPU Implementation of a Nonlinearly Implicit One-Dimensional Particle-in-Cell Algorithm. Journal of Computational Physics, 231, 5374-5388. https://doi.org/10.1016/j.jcp.2012.04.040
Azevedo, E.F.D., et al. (2013) Hybrid MPI/OPENMP/GPU Parallelization of XGC1 Fusion Simulation Code. The International Conference for High-Performance Computing, Networking, Storage and Analysis, Denver, CO, 17-22 November.
Ibrahim, K.Z., et al. (2013) Analysis and Optimization of Gyrokinetic Toroidal Simulations on Homogenous and Heterogenous Platforms. The International Journal of High Performance Computing Applications, 27, 454-473. https://doi.org/10.1177/1094342013492446
Decyk, V.K. and Singh, T.V. (2014) Particle-in-Cell Algorithms for Emerging Computer Architectures. Computer Physics Communications, 185, 708-719. https://doi.org/10.1016/j.cpc.2013.10.013
Pang, X. and Rybarcyk, L. (2014) GPU Accelerated Online Multi-Particle Beam Dynamics Simulator for Ion Linear Particle Accelerators. Computer Physics Communications, 185, 744-753. https://doi.org/10.1016/j.cpc.2013.10.033
Shah, H., Kamaria, S., Markandeya, R., Shah, M. and Chaudhury, B. (2017) A novel Implementation of 2D3V Particle-in-Cell (PIC) Algorithm for Kepler GPU Architecture. Proceedings of 2017 IEEE 24th International Conference on High-Performance Computing (HiPC), 1, 378-387.
Fatemi, S., Poppe, A.R., Delory, G.T. and Farrell, W.M. (2017) AMITIS: A 3D GPU-Based Hybrid-PIC Model for Space and Plasma Physics. Journal of Physics: Conference Series, 837, Article ID: 012017. https://doi.org/10.1088/1742-6596/837/1/012017
Titan (2019) Titan Advancing the Era of Accelerated Computing. https://www.olcf.ornl.gov/olcf-resources/compute-systems/titan/
Hockney, R.W. and Eastwood, J.W. (1988) Computer Simulation Using Particles. Adam Hilger, New York.
Netzer, R.H. and Miller, B.P. (1992) What Are Race Conditions?: Some Issues and Formalizations. ACM Letters on Programming Languages and Systems (LOPLAS), 1, 74-88. https://doi.org/10.1145/130616.130623
Quinn, M.J. (2004) Parallel Programming in C with MPI and Open MP. McGraw-Hill Inc., New York.
Qiang, J. and Li, X. (2010) Particle-Field Decomposition and Domain Decomposition in Parallel Particle-in-Cell Beam Dynamics Simulation. Computer Physics Communications, 181, 2024-2034. https://doi.org/10.1016/j.cpc.2010.08.021
Nvidia, C. (2010) Cufft Library.
Tiwari, D., Gupta, S., Gallarno, G., Rogers, J. and Maxwell, D. (2015) Reliability Lessons Learned from GPU Experience with the Titan Supercomputer at Oak Ridge Leadership Computing Facility. In: Proceedings of the International Conference for High-Performance Computing, Networking, Storage and Analysis, ACM, New York, 38. https://doi.org/10.1145/2807591.2807666
Wells, J., Bland, B., Nichols, J., Hack, J., Foertter, F., Hagen, G., Maier, T., Ashfaq, M., Messer, B. and Parete-Koon, S. (2016) Announcing Supercomputer Summit. Technical Report, ORNL (Oak Ridge National Laboratory), Oak Ridge, TN.