ImplicitGlobalGrid.jl

Appearance

ImplicitGlobalGrid.jlDistributed parallelization of stencil-based GPU and CPU applications

Close to ideal weak scaling on thousands of GPUs.

Usage 🚀
API reference 📚
View on GitHub
ImplicitGlobalGrid.jl
🧩

Three functions

init_global_grid, update_halo! and finalize_global_grid can be enough to turn a single-GPU/CPU code into a massively scaling Multi-GPU/CPU application.

🌐

Implicit global grid

The global grid is created implicitly from the number of processes and the process topology, defined automatically or chosen by the user; 1-D, 2-D and 3-D grids are supported.

🖥️

GPUs and CPUs

Nvidia and AMD GPUs through CUDA.jl and AMDGPU.jl, with automatic device selection, and CPUs with multi-threaded halo copies through Polyester.jl.

⚡

GPU-aware MPI

Halo updates close to hardware limit with MPI.jl; CUDA-aware or ROCm-aware MPI is activated with an environment variable, and communication can be hidden behind computation.

🔗

Interoperable with MPI.jl

The Cartesian communicator is exposed, the initialization and finalization of MPI can be left to the application, and every argument passed to MPI.jl is customizable.

📐

Global coordinates and extents

Global coordinates, sizes and indices of the local grid, and local and global array extents, e.g. for parallel VTK output.

📊

In-situ visualization

gather! assembles an array from all processes on a single process according to the global grid, for in-situ visualization or monitoring.

🗂️

Multiple grids

Several global grids can be created and activated; update_halo! operates on the active grid or on the grid passed to it.

🧊

Arrays of small arrays / structs

update_halo! supports the logical arrays of small arrays or structs of CellArrays.jl.

What is ImplicitGlobalGrid.jl? ​

ImplicitGlobalGrid is an outcome of a collaboration of the Swiss National Supercomputing Centre, ETH Zurich (Dr. Samuel Omlin) with Stanford University and the Geocomputing Centre, University of Lausanne (Dr. Ludovic Räss and Prof. Yuri Podladchikov). It renders the distributed parallelization of stencil-based GPU and CPU applications on a regular staggered grid almost trivial and enables close to ideal weak scaling of real-world applications on thousands of GPUs.

The global computational grid is created implicitly from the number of processes the application is run with: the user writes a code for one GPU/CPU (local grid); then, as little as three functions can be enough to transform it into a massively scaling Multi-GPU/CPU application. Halo updates rely on MPI.jl and can leverage CUDA-aware or ROCm-aware MPI. 1-D, 2-D and 3-D grids are supported. See the Overview for more details.

Installation ​

ImplicitGlobalGrid can be installed directly with the Julia package manager from the Julia REPL:

julia
import Pkg; Pkg.add("ImplicitGlobalGrid")

Dependencies ​

ImplicitGlobalGrid relies on the Julia MPI wrapper (MPI.jl) and CellArrays.jl; GPU support is provided through package extensions for the Julia CUDA package (CUDA.jl [1, 2]) and the Julia AMDGPU package (AMDGPU.jl), and multi-threaded CPU halo copies through Polyester.jl.

Contributors ​

The principal contributors to ImplicitGlobalGrid.jl are (ordered by the significance of the relative contributions):

  • Dr. Samuel Omlin (@omlins), CSCS (Swiss National Supercomputing Centre), ETH Zurich

  • Dr. Ludovic Räss (@luraess), Geocomputing Centre, University of Lausanne

  • Dr. Ivan Utkin (@utkinis), Laboratory of Hydraulics, Hydrology, Glaciology, ETH Zurich

Questions, comments and discussions ​

To discuss technical issues, please post on Julia Discourse in the GPU topic or the Julia at Scale topic or in the #gpu or #hpc channels on the Julia Slack. To discuss numerical/domain-science issues, please post on Julia Discourse in the Numerics topic or the Modelling & Simulations topic or whichever other topic fits best your issue.

Contributing ​

Contributions are welcome, be it new examples, convenience features or core improvements! Please open an issue to discuss your idea beforehand. A pull request should address an issue in its completeness and blend nicely into the existing project (community guidelines such as ColPrac can serve as inspiration).

Citing ​

If you use ImplicitGlobalGrid in your work, please cite [3]. The BibTeX entry and further publications are found in the References.

Funding ​

The development of this package is supported by the GPU4GEO and ∂GPU4GEO PASC projects. More information about the GPU4GEO project can be found on the GPU4GEO website.