Usage

Multi-GPU with three functions

Only three functions are required to perform halo updates close to hardware limit:

• init_global_grid

• update_halo!

• finalize_global_grid

Three additional functions are provided to query Cartesian coordinates with respect to the global computational grid if required:

• x_g

• y_g

• z_g

Moreover, the following three functions allow to query the size of the global grid:

• nx_g

• ny_g

• nz_g

The 50-lines Multi-GPU example illustrates how these functions enable the creation of massively parallel applications.

Further tools: global indices (ix_g, iy_g, iz_g), local and global extents of arrays, e.g. for parallel VTK output (extents, extents_g, metagrid) and timing (tic, toc).

GPU support and device selection

Support for Nvidia or AMD GPUs is activated by importing CUDA.jl or AMDGPU.jl before ImplicitGlobalGrid:

CPUs

using ImplicitGlobalGrid

Nvidia GPUs

using CUDA
using ImplicitGlobalGrid

AMD GPUs

using AMDGPU
using ImplicitGlobalGrid

By default, init_global_grid selects the GPU corresponding to the node-local MPI rank (see select_device); every rank must then see all GPUs of its node. If the job launcher already binds one GPU per rank (e.g. Slurm --gpus-per-task=1), set select_device=false. To run on CPUs only while a GPU package is loaded, set device_type="none".

CUDA-/ROCm-aware MPI support

If the system supports CUDA-aware MPI (for Nvidia GPUs) or ROCm-aware MPI (for AMD GPUs), it can be activated for ImplicitGlobalGrid by setting the following environment variable (at latest before the call to init_global_grid):

Nvidia GPUs

export IGG_CUDAAWARE_MPI=1

AMD GPUs

export IGG_ROCMAWARE_MPI=1

It can also be activated per dimension, see Environment variables.

Multi-threaded CPU halo copies

Halo copies of CPU arrays can be multi-threaded with Polyester.jl by importing it before ImplicitGlobalGrid, setting IGG_USE_POLYESTER=1 and starting Julia with multiple threads (e.g. julia -t 4):

using Polyester
using ImplicitGlobalGrid

CellArrays

update_halo! also accepts CellArrays of CellArrays.jl (with blocklength 0 or 1).

Multiple grids

init_global_grid creates and activates one grid. Additional grids are created with create_global_grid and selected with activate_global_grid (or passed to update_halo! with the keyword active_global_grid); get_global_grid returns the active grid:

init_global_grid()                          # Initialize the package without creating a grid.
gg1 = create_global_grid(nx, ny, nz)
gg2 = create_global_grid(2nx, 2ny, 2nz)
activate_global_grid(gg1)
update_halo!(A1)                            # Uses gg1.
update_halo!(A2; active_global_grid=gg2)    # Uses gg2.

In-situ visualization / monitoring

ImplicitGlobalGrid provides a function to gather an array from each process into a one large array on a single process, assembled according to the global grid:

• gather!

This enables straightforward in-situ visualization or monitoring of Multi-GPU/CPU applications as shown in this example.

Seamless interoperability with MPI.jl

ImplicitGlobalGrid is seamlessly interoperable with MPI.jl. The Cartesian MPI communicator it uses is created by default when calling init_global_grid and can then be obtained as follows (variable comm_cart):

me, dims, nprocs, coords, comm_cart = init_global_grid(nx, ny, nz);

Moreover, the automatic initialization and finalization of MPI can be deactivated in order to replace them with direct calls to MPI.jl:

init_global_grid(nx, ny, nz; init_MPI=false);

finalize_global_grid(;finalize_MPI=false)

Besides, init_global_grid makes every argument it passes to an MPI.jl function customizable via its keyword arguments.

Environment variables

The following environment variables are read when a grid is created (init_global_grid or create_global_grid); a value > 0 activates the feature:

Variable	Description
IGG_CUDAAWARE_MPI	Use CUDA-aware MPI in all dimensions.
IGG_CUDAAWARE_MPI_DIMX, IGG_CUDAAWARE_MPI_DIMY, IGG_CUDAAWARE_MPI_DIMZ	Use CUDA-aware MPI in the given dimension (only read if IGG_CUDAAWARE_MPI is not activated).
IGG_ROCMAWARE_MPI	Use ROCm-aware MPI in all dimensions.
IGG_ROCMAWARE_MPI_DIMX, IGG_ROCMAWARE_MPI_DIMY, IGG_ROCMAWARE_MPI_DIMZ	Use ROCm-aware MPI in the given dimension (only read if IGG_ROCMAWARE_MPI is not activated).
IGG_USE_POLYESTER	Use Polyester for the CPU halo copies in all dimensions (requires importing Polyester).
IGG_USE_POLYESTER_DIMX, IGG_USE_POLYESTER_DIMY, IGG_USE_POLYESTER_DIMZ	Override IGG_USE_POLYESTER in the given dimension (only read if IGG_USE_POLYESTER is activated).