Navigation and service

Space weather simulation

Space weather simulation

Simulating and understanding space weather phenomena is of critical importance for the safety of our electrical, telecommunications, and space infrastructure. Within the DEEP project, KU Leuven works with a Particle-in-Cell (PIC) code to understand the effects of the plasma that constantly flows from the Sun to the Earth. iPiC3D is a widely used code routinely run on NASA's Pleiades supercomputer and allows studying a large portion of the system while retaining the full detailed microphysics. The main challenge in the Space Weather Simulations is the presence of multiple temporal and spatial scales. Today's Petascale computers do not allow to reach the full scale of the processes required to be simulated due to lack of memory to store all particles and insufficient compute power.

On one hand, Particle-in-Cell codes are fundamentally massively parallel, since the very large number of charged particles required for the simulation can be processed independently. On the other hand, iPiC3D couples the particle move with a field solver, which requires constant communications, limiting its scalability. As a result, the particle solver is placed in the Booster, whereas the field solver runs on the Cluster. The amount of data transferred between the solvers is small and limited to the field information stored in the cartesian grid from the field and particle solvers. This allows to decouple the requirements of both parts of the application, hence allowing to increase the number of particles per cell and therefore reduce the noise of the simulation.

For optimal performance the particle processing had to be optimised for Xeon Phi processors. The code evolved from a pure MPI implementation to a hybrid MPI+OpenMP implementation, focusing in particular on the computation of the particle movement and its associated fields. Physical requirements of the algorithm push towards an array of structs (AoS) data layout, to allow for an easy sorting of particles. However, to achieve efficient vectorization, a fast on-the-fly transposition from AoS to SoA was implemented using intrinsics. The overall result is an extra edge of performance and accuracy using the DEEP architecture.

"Space weather provides an ideal testbed for the design and experimentation of exascale architectures because two elements form a typical space weather simumation: fields and particles simulation. To increase the fidelity and predictive capability, it is necessary to increase the number of particles and the number of points where the fields are described. The particles evolve as single units and are highly scalable in a parallel hardware, but the fields are highly interconnected and require more communication, providing a greater challenge to parallel scaling and limiting the performance of the whole application when running in tens or hundreds of thousands of cores. DEEP provides a unique opportunity: to take full benefit of the scalable particles on the Booster, while keeping the less computing intensive fields in the Cluster. The particles move and can migrate thanks to the communication between Booster nodes, and communicate the changes in the induced electromagnetic fields to the Cluster. This feature makes DEEP unique and truly interesting for space weather." - Giovanni Lapenta, KULeuven