_spotlights/_picongpu.md

----
-layout: spotlight
-
-# Spotlight list attributes
-name: PIConGPU
-preview_image: picongpu/picongpu-logo.png
-excerpt: PIConGPU is a relativistic Particle-in-Cell code running on graphic processing units. It is Open Source und is freely available for download. PIConGPU is developed and maintained by the Junior Group Computational Radiation Physics at the Institute for Radiation Physics at HZDR in close collaboration with the Center for Information Services and High Performance Computing (ZIH) of the Technical University Dresden.
-
-# Title for individual page
-title_image:
-title: PIConGPU - A Many-GPGPU Particle-in-Cell Code
-keywords:
-    - GPU
-    - Particle-in-Cell
-    - Simulation
-hgf_research_field: Matter
-hgf_centers:
-    - HZDR
-contributing_organisations:
-    - TU Dresden
-scientific_community:
-    - Matter / Photon Science
-impact_on_community:
-contact: m.bussmann@hzdr.de
-platforms:
-    - type: github
-      link_as: https://github.com/ComputationalRadiationPhysics/picongpu
-    - type: webpage
-      link_as: https://picongpu.readthedocs.io/
-license: GPL-3.0-only
-costs:
-software_type:
-    - Simulation
-application_type:
-    - Desktop
-programming_languages:
-    - C++
-doi: 10.1109/TPS.2010.2064310
-funding:
----
-
-# PIConGPU Features
-
-* A variable, Yee-like grid, on which electric and magnetic fields are approximated.
-* Particle-Pushers following the works of Boris and Vay
-* Maxwell-Solvers as proposed by Yee and Lehe
-* Current-Deposition schemes published by Esirkepov and Villasenor-Buneman
-* Macro-Particle Form Factors: NGP, CIC, TSC, PSQ
-* Computation of far field radiation due to electron motion
-* Parallel HDF5 Output
-* Online-Visualization (2D/3D)
-* The Particle-in-Cell algorithm is a central tool in plasma physics. It describes the dynamics of a plasma by computing the motion of electrons and ions in the plasma based on Maxwell's equations.
-
-## How does the Particle-in-Cell Algorithm work?
-
-The PIC algorithm solves the so-called Maxwell-Vlasov equation. To solve this equation, electric and magnetic fields are interpolated on a physical grid dividing the simulated volume into cells.
-
-Charged particles like electrons and ions are modeled by macro-particles. These can describe the motion of up to several hundred particles by the motion of a single spread-out particle distribution. The macro-particles' motion is influenced by the electric and magnetic fields on the grid.
-
-The particle motion in turn creates currents. Following Ampère's law these currents create magnetic fields. These magnetic fields in turn create electric fields as described by Faraday's law.
-
-These new fields then act back on the particles.
-
-## What is so new about PIConGPU?
-
-GPUs show very high computational performance, because many processors work in parallel. In order to make the most out of this performance, the processors should work independently of each other. In case of the PIC-Algorithm this is hard to achieve, since in the current deposition step, currents which are fixed to the cells have to be computed from the velocity of particles moving freely between grid cells. This motion leads to memory access patterns in which current data and particle data are located at different places in the memory and parallel processes can disturb each others execution when accessing the same part of the memory.
-
-Recently, this problem was solved in our group using a new data model for particle and grid-based data and asynchronous data transfer.
-
-## All that with a single GPU?
-
-No, because GPUs do not have enough memory to simulate large physical systems. This makes it necesarry to use more than one GPU and distribute the simulated volume between the GPUs.
-
-The problem with this approach is the data transfer between GPUs. For this, data has to be transferred from the GPU to the main memory of the computer housing the GPU. The data has then to be sent via network to the other computers housing the other GPUs
-
-This process normally takes a long time and GPUs have to wait for the end of the data transfer before continuing their computations.
-
-We were able to solve this problem by interleaving the data transfer between GPUs and the computation on a single GPU, so that the GPUs can execute the algorithmic steps continuously without interruption [3,4]. This was only possible because we got help from ZIH, TU Dresden, which provided an efficient library for data transfer between GPUs and tools to measure the performance of our code.
-
-## What does this mean for simulations?
-
-We want to speed up the time of the simulation to reduce the time between the start of the simulation and the reception of the final result. With GPUs this speed up can mean that a simulation that normally takes a week to finish can finish within a few hours.
-

_spotlights/picongpu.md

+---
+layout: spotlight
+
+# Spotlight list attributes
+name: PIConGPU
+preview_image: picongpu/picongpu-logo.png
+excerpt: PIConGPU is a relativistic Particle-in-Cell code running on graphic processing units as well as regular multi-core processors. It is Open Source und is freely available for download. PIConGPU is developed and maintained by the Computational Radiation Physics Group at the Institute for Radiation Physics at HZDR in close collaboration with the Center for Advanced Systems Understanding (CASUS).
+
+# Title for individual page
+title_image:
+title: PIConGPU - A Many-GPGPU Particle-in-Cell Code
+keywords:
+    - GPU
+    - Particle-in-Cell
+    - Simulation
+hgf_research_field: Matter
+hgf_centers:
+    - HZDR
+contributing_organisations:
+    - Center for Advanced Systems Understanding
+    - University of Delaware
+    - Oak Ridge National Laboratory
+scientific_community:
+    - Matter / Photon Science
+impact_on_community:
+contact: m.bussmann@hzdr.de
+platforms:
+    - type: github
+      link_as: https://github.com/ComputationalRadiationPhysics/picongpu
+    - type: webpage
+      link_as: https://picongpu.readthedocs.io/
+license: GPL-3.0-only
+costs: free and open
+software_type:
+    - Simulation
+application_type:
+    - Desktop
+programming_languages:
+    - C++
+doi: 10.1109/TPS.2010.2064310
+funding:
+     name: HZDR, CASUS, ORNL CAAR project
+---
+
+# PIConGPU 
+
+[PIConGPU](https://github.com/ComputationalRadiationPhysics/picongpu) is an extremely scalable and platform portable application for particle-in-cell simulations. While we mainly use it for studying laser-plasma interactions, it has also been used for astrophysics studies of the Kelvin-Helmholtz-instability.
+
+PIConGPU has been a finalist for the prestigious Gordon-Bell-Award in 2013 and has been one of the flagship applications for a number of leading edge high performance computing (HPC) systems since then ([Titan](https://www.olcf.ornl.gov/2018/07/17/titan-helps-scientists-fine-tune-laser-interactions-to-advance-cancer-treatments/), [JUWELS Booster](https://www.fz-juelich.de/portal/EN/Research/ITBrain/Supercomputer/early_access_app/tb_picongpu.html?nn=2692138), [Frontier1](https://www.amd.com/system/files/documents/ornl-picongpu-team-aims-amd-instinct-gpus-at-laser-particle-accelerator-development.pdf), [Frontier2](https://www.olcf.ornl.gov/2021/01/12/university-of-delaware-team-tightens-up-code-for-exascale-computing-on-frontier/), [Frontier3](https://ascr-discovery.org/2021/12/shrinking-big-physics/)).
+Through this work the PIConGPU has established strong ties with a lot of national and international partners, especially the underlying hardware agnostic libraries like [Alpaka](https://github.com/alpaka-group/alpaka) and [Llama](https://github.com/alpaka-group/llama) are now adopted in the CERN LHC software stack as well. Another collaborative effort also driven by PIConGPU is a standardization in data formats for plasma physics via [openPMD](https://github.com/openPMD/), which is becoming one of the leading data standard in the community.
+
+<div class="spotlights-text-image">
+<img src="{{ site.directory.images | relative_url}}spotlights/picongpu/LWFAVectorFieldVisTest.jpg" alt="A snapshot from a test simulation of an ultrashort, high-intensity laser pulse.">
+<span>A snapshot from a test simulation of an ultrashort, high-intensity laser pulse (orange-striped sphere) driving a plasma wave in ionized helium gas on the Oak Ridge Leadership Computing Facility’s (OLCF) Summit supercomputer. Purple areas highlight the electron density. Streams depict the stronger (red) and weaker (green and blue) electric fields. See also
+<a href="https://www.youtube.com/watch?v=yPq22Lz_kb8"><i class="fas fa-external-link-alt"></i> video on this</a>
+(<i>Link to Youtube</i>).
+<i>This image was generated using ISAAC, a tool for visualizing simulations in real time on the  Frontier supercomputer being built at OLCF. Image Courtesy of Felix Meyer/Helmholtz-Zentrum Dresden-Rossendorf.</i></span>
+</div>
+
+## Features
+
+* Open source, fully relativistic, 3D3V, many-core, performance portable Particle-in-cell code with a single code base.
+* A variable, Yee-like grid, on which electric and magnetic fields are approximated.
+* Particle-pushers following the works of Boris and Vay
+* Maxwell-solvers as proposed by Yee and Lehe, as well as an arbitrary-order FDTD field solver
+* Current-deposition schemes according to Esirkepov, Villasenor-Buneman and ZigZag
+* Macro particle shape orders: NGP (0th) to P4S (4th)
+* In-situ computation of coherent and incoherent classical far-field radiation due to charged-particle motion
+* Self-consistent additions to the PIC cycle, e.g.:
+	- QED synchrotron radiation and Bremsstrahlung (photon emission)
+	- Thomas-Fermi collisional ionization
+	- ADK and BSI field ionization
+	- Classical radiation reaction
+* Scalable, parallel I/O for restarts and full output via [openPMD](https://github.com/openPMD/) to HDF5 and ADIOS2
+* Hardware abstraction for compute target platforms through [Alpaka](https://github.com/alpaka-group/alpaka)
+* Online-Visualization (2D/3D) via [ISAAC](https://github.com/ComputationalRadiationPhysics/isaac)
+* Extensible selection of plugins for online analysis of particle and field data
+
+## How does the Particle-in-Cell Algorithm work?
+
+The PIC algorithm approximates the solution of the so-called Maxwell-Vlasov equation. In this approach electric and magnetic fields are interpolated on a physical grid dividing the simulated volume into cells.
+
+Charged particles like electrons and ions are modeled by macro-particles. Each of these describe the motion of up to several hundred real particles with same momentum by the motion of a single spatially spread-out particle distribution. The macro-particles' motion is influenced by the electric and magnetic fields on the grid.
+
+The particle motion in turn creates currents. Following Ampère's law these currents create magnetic fields, which then are used to compute the electric fields as described by Faraday's law.
+
+These new fields then act back on the particles.
+
+## What is so new about PIConGPU?
+
+GPUs (Graphics processing units) show very high computational performance, because many processors work in parallel. In order to make the most out of this performance, the processors should work independently of each other. In case of the PIC-Algorithm this is hard to achieve, since in the electrical current deposition step, currents which are fixed to the cells have to be computed from the velocity of particles moving freely between grid cells. This motion leads to memory access patterns in which electrical current data and particle data are located at different places in the memory and parallel processes can disturb each others execution when accessing the same part of the memory.
+This problem was solved in our group using a novel data model for particle and grid-based data and asynchronous data transfer.
+
+Furthermore, PIConGPU is written in a hardware agnostic way using an hierarchical system model, that is mapped to the available compute resources by the zero-overheap compile time library [Alpaka](https://github.com/alpaka-group/alpaka). As a result PIConGPU shows similarly great performance on all supported platforms (CPUs from all vendors, as well as AMD and NVIDIA GPUs and FPGAs).
+
+## All that with a single GPU?
+
+No, because GPUs do not have enough memory to simulate large physical systems. This makes it necessary to use more than one GPU and distribute the simulated volume between the GPUs.
+
+The problem with this approach is the data transfer between GPUs. For this, data has to be transferred from the GPU to the main memory of the computer housing the GPU. The data has then to be sent via network to the other computers housing the other GPUs
+
+This process normally takes a long time and GPUs have to wait for the end of the data transfer before continuing their computations.
+
+We were able to solve this problem by interleaving the data transfer between GPUs and the computation on a single GPU, so that the GPUs can execute the algorithmic steps continuously without interruption.
+
+## What does this mean for simulations?
+
+We want to speed up the time of the simulation to reduce the time between the start of the simulation and the reception of the final result. With GPUs this speed up can mean that a simulation that normally takes a week to finish can finish within a few hours.
+